Word list by frequency based on Open Subtitle corpus 2018

A while ago I started working on newer version of Word list based 2018 OpenSubtitle Corpus. Downloading the data takes a while and can be frustrating – Microsoft Edge not being able to continue down in the background (should you close the main window) etc. I got distracted into writing a download manager.

Thanks To Hugo Lopez‘s comments on GitHub I was reminded again and I got started the 2nd time.

Based on comments from Hugo and others I added Language DetectionLanguage Detection to the list generator and generated the word lists.

I have not sanitised the lists in any way.
50k wordlists were only created only if overall word count exceeded 50k
Created ignored file that lists words identified as invalid. It is very likely that many words inside the ignored files are valid. Their frequency within the corpus is preserved should you wish to correct them.
With ja and fr a few input files contained invalid words. I added check to prevent the generator from crashing. Further processing of that file was not done.
From a given directory, only 1 file was processed. Any additional files in the input data were ingored.
The generated dataset can be found on Github along with code.

https://github.com/hermitdave/FrequencyWords
https://github.com/hermitdave/FrequencyWords/tree/master/content/2018

Advertisement

Single Page Application with #uwpdev

Over the next few days I will push a set of blog posts detailing the design of Daily Mail Online app for Windows. This has been due for at least 12 months now.

I am not going to discuss the Content published by the App rather the App in itself – I am the creator of the App and have no control over the content.

Traditionally most applications have multiple views / pages. User is presented with different pages depending upon the data to be shown. Page transition theme makes navigation between pages a joy.

I had issues with page navigation – especially when the page you are navigating to contains a FlipView control and the item to be displayed is anything but the first item – there is an obvious rendering issue where the first item renders momentarily before the correct item is shown – really makes me want to kick someone in the nuts every time I see it.

During my rework of Daily Mail Online app for desktop last year, I decided to go the SPA route. SPA stands for Single Page Application. Whilst it has been around for a very long time, I first heard it from Benjamin Howarth when he did a talk at London Mobile Dev user group back in 2015

So the idea of SPA is simple
* Get all necessary data
* Display correct view depending upon what the user desires.
* Maintain pseudo navigation to maintain sanity.

Daily Mail Online app has the following views:
Channel view
Article view
Related Article View
Comment View
Topic View
Search View
Profile View
Read Later View
Photo Gallery View
Video View

In early days of UWP, one could use Deferred Loading but there was no way to unload anything you didn’t want. That meant that from a Visual Tree point of view it was messed up – all these views – all rendered and in memory just hidden away when not needed.

Last year with Creators update, Windows Dev team introduced a feature called x:Load. It allows dynamically loading and unloading of controls – providing an experience similar to visibility toggle while having a lighter footprint.

<ListView
    x:Name="ArticleListView"
    Grid.Row="1"
    Grid.RowSpan="2"
    x:Load="{x:Bind MainViewModel.LoadSingleArticleView, Mode=OneWay}"
    Background="{ThemeResource ContentBackgroundBrush}"
    BorderBrush="{ThemeResource ContentBackgroundBrush}"
    BorderThickness="0"
    ItemContainerStyle="{StaticResource GroupItemStyle}"
    ItemTemplateSelector="{StaticResource ArticleLayoutSelector}"
    ItemsSource="{Binding ElementName=mainPage, Path=DataContext.RelatedItemContent, Mode=OneWay}"
    Loaded="OnArticleListViewLoaded"
    SelectionMode="None"
    ShowsScrollingPlaceholders="False"
    VirtualizingStackPanel.VirtualizationMode="Recycling" />

The only requirement is that you need to Name the control and Laoding / Unloading can be done by binding to VM property – it can’t really get any simpler.

Along with SPA design, I use ReactiveUI along with traditional MVVM within the Daily Mail Online App. More on injecting ReativeX goodness within MVVM app in next post.

End of Cineworld app for #Windows and issues facing indie dev #uwpdev

Back in the summer of 2016, I sat waiting for a movie to start at Cineworld. Wanted to check something, I was forced to use the mobile website. At that point I wondered if they had an API and within 2 days I had a version of the app ready for Windows Phone store.

A few months down the line, I ported it to Windows 8 and then ended up using Azure to generate dataset for the apps.  For the last 2 years or so I have been paying for the service.. it wasn’t significant but still odd 200 a month and having looked at IAP purchase in the last 2 months I decided that it was time to call it a day.

While Microsoft might say mobile didn’t work for them, I’d beg to differ. Windows Mobile version of the app had nearly 4k downloads (3976) vs desktop app with 256 downloads. YES a full 256 downloads.

Mobile app – Windows Phone app (WP8x and Windows 10 Mobile)

Desktop app – Windows 8 app (Windows 8x and Windows 10 Desktop)

That tells you all you need to know about consumption of certain apps on desktop vs mobile – it might not be global but it definitely is so in UK.

Now lets look at IAP.

IAP for Mobile app

IAP for Desktop app

Above are for the last 2 months – minus governmental and store taxation, I could have paid for the Azure service for a month.

Having decided that it wasn’t economical, the fact that Microsoft officially acknowledged end of the road for mobile was sufficient to determine that the app will never ever pay for itself.

The source for the Apps and Azure services is available on GitHub

Most code was before I adopted MVVM for Windows Development so please except my apologies for whatever you find in there.

Happy coding

Operating system analysis of a small #uwp app user base #uwpdev

In 2015, I released Windows Phone 8.1 and Windows 10 version of the Daily Mail Online app. The apps uses tracking to monitor usage etc. In the last set of builds for the first iteration of my work there, I set the OS to be Windows Phone 8.1, Windows 10 and Windows 10 Mobile. Whilst it was sufficient back then, it meant that I did not have sufficient data to set the Min Target SDK the 2nd time around.

Let me start by saying that my data is mostly UK centric.

Such data hasn’t so far been released by Microsoft and to ensure that I targeted as many users as possible I settled for a min target SDK to be 10240 (the first version of Windows 10).

I still have users on the early version of the app and those will never upgrade. There are significant users on Windows Phone 8.1 (dropping at a steady rate)

I have been monitoring this numbers every day and I finally decided to modify the Min SDK to 14939. This is why.

Of the Windows 10 users, majority were on 14393. 0.7% users are still on 10240, around 1.9% are on 10586.

I found 16 different builds of 10240 in use amongst those 0.7%

21 different builds for 10586

With number of Insiders matching those on older generations, its time move forward. Min target for next update is 14393 and a total of 33 Windows 10 builds to support.

Beware oh beware of the Nullable type #uwp

Having done limited rollout yesterday I noticed a strange pattern. A huge number of crashes with “System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw”

The most interesting bit was that the failures were only mostly on phones and not on desktop devices.

What the heck!! looking inside didn’t help much. The crash group on HockeyApp also mentioned ‘invalidoperation_novalue‘ Okay this was strange.. In another instance I had a similar invalidoperation_novalue but somewhat different location ‘System.Nullable$1.get_Value ‘

In a few places I was using nullable int. that translates to System.UInt32

int? pos = null;

// elsewhere read it
var item = list[pos.HasValue ? pos.Value : 0];

// in other places I encountered some exceptions - can't remember what and I used null check rather than HasValue
var item2 = list[pos != null ? pos.Value : 0]; 

//or in newer iteration of csharp
var item3 = list[pos?.Value??0];

Somehow I was directly using instance of nullable directly. A few posts helped. I was sure that was the problem. Thanks to Carsten Schuette for showing the use of GetValueOrDefault

InvalidOperationException in DeviceUtils.DiagonalSize
System.InvalidOperationException must contain value for ViewModel

So in the latest build, I have replaced above usage with

int? pos = null;

// elsewhere read it
var item = list[pos.GetValueOrDefault(0)];

Only hoping that I identified and fixed the one that was causing the issue. Stack trace with .NET Native can be painful.

Exception Type:  System.InvalidOperationException
Crashed Thread:  2

Application Specific Information:
InvalidOperation_NoValue. For more information, visit http://go.microsoft.com/fwlink/?LinkId=623485

Exception Stack:
SharedLibrary!<BaseAddress>+0x388719
MOL-Desktop!<BaseAddress>+0x121e122
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
SharedLibrary!<BaseAddress>+0x353fa2
SharedLibrary!<BaseAddress>+0x353a0e
SharedLibrary!<BaseAddress>+0x38387a
MOL-Desktop!<BaseAddress>+0x121d783
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
SharedLibrary!<BaseAddress>+0x3c2c09
SharedLibrary!<BaseAddress>+0x35369a
MOL-Desktop!<BaseAddress>+0xf08a62
MOL-Desktop!<BaseAddress>+0xf0b397
MOL-Desktop!<BaseAddress>+0xf0b2ef
Microsoft.HockeyApp.Extensibility.Windows.UnhandledExceptionTelemetryModule.CoreApplication_UnhandledErrorDetected(Object sender, ApplicationModel.Core.UnhandledErrorDetectedEventArgs e)

Thread 2 Crashed:
0   SharedLibrary                        0x0000000002908719 0x0000000002580000 + 3704601
1   MOL-Desktop                          0x0000000001e1e122 0x0000000000c00000 + 18997538
2   SharedLibrary                        0x00000000028d3fcc 0x0000000002580000 + 3489740
3   SharedLibrary                        0x00000000028d3fa2 0x0000000002580000 + 3489698
4   SharedLibrary                        0x00000000028d3a0e 0x0000000002580000 + 3488270
5   SharedLibrary                        0x000000000290387a 0x0000000002580000 + 3684474
6   MOL-Desktop                          0x0000000001e1d783 0x0000000000c00000 + 18995075
7   SharedLibrary                        0x00000000028d3fcc 0x0000000002580000 + 3489740
8   SharedLibrary                        0x0000000002942c09 0x0000000002580000 + 3943433
9   SharedLibrary                        0x00000000028d369a 0x0000000002580000 + 3487386
10  MOL-Desktop                          0x0000000001b08a62 0x0000000000c00000 + 15764066
11  MOL-Desktop                          0x0000000001b0b397 0x0000000000c00000 + 15774615
12  MOL-Desktop                          0x0000000001b0b2ef 0x0000000000c00000 + 15774447
13  MOL-Desktop                          0x0000000001dd44db 0x0000000000c00000 + 18695387

2017 – MVP award – Great Start

On 1st of Jan 2017 I got a mail from Microsoft stating that I have been awarded an MVP award for my community contributions for Windows Development.

I am honoured to have been nominated and to have been awarded this. For a developer in Microsoft technologies this is the highest honour (sort of). Thank you @WindowsDev and @MVPAward

I think 2016 was a mixed bag and technically I did little to have deserved this honour. I have been a contributor to the UWP Community Toolkit and that was the primary driver. I am sure I did substantially more from 2011 – 2015 but hey maybe that’s what it took 🙂 still feels surreal

Going ahead I will try to blog more about #UWP and #UWPDev in 2017. Happy new year all

Old photos

Json Formatter Edge Extension #Windows10

The only reason I install Chrome now and again is to use extensions like Json Formatter extension. Recently Microsoft released Windows 10 Anniversary update.
Edge browser on this update supports extensions. This list of extensions on store is still limited. The team is analysing the performance and publishing them accordingly.

Can’t wait any longer. I started by cloning Json-Formatter repo and used Microsoft Edge Extension Toolkit to port the extension to work on Edge.

You can download the repo and install the extension after enabling Extension developer mode from ‘about:flags’

The repo is on Github https://github.com/hermitdave/json-formatter-edge along with instructions

An example geonames json service can be found here

By default the json is displayed like this on Edge browser.

With the extension installed, it is displayed at below

To port another extension use the Micosoft Edge Extension Toolkit available on Store

What to do when your Surface devices don’t sleep

Starting Windows 8 Microsoft introduced a sleep state called Connected Standby (CS). Think of this sleep as your smartphone when its screen is off, the device is still running, connected to the mobile networks, capable of receiving notifications and wakeup calls etc.

In early days this feature was supported only on WOA… you know it as Windows RT.. yes something many wouldn’t care to remember.. Windows x86 / x64 SKU officially support CS starting with Haswell processor. Haswell introduced a sleep state S0. This keeps part of the device sufficiently awake to receive notifications etc while in low power state.

It never truly worked for me. I had one of the first engineering models from Intel delivered in June 2013 and the device refused to wake up properly. I tried just about everything and while things have gotten better, for people like myself, battery life is more important than notification (which I already get on my phones). A few times SP4 in CS has had around 50 – 90% drain in few hours ( 2 – 4).

The solution is easy – unless CS is something you rather than instead of longer battery life, you should force system to hibernate rather than go into connected standby.

Switching to hibernate:

• press Start Menu and type power options
• open Power Options from Control Panel.
• tweak Choose what power buttons do option – enable hibernation and set desired settings
• tweak Change when the computer sleeps – I prefer a short sleep
• next click Change advanced power settings
• Navigate to Sleep > Hibernate After > Ensure duration is correct.
• Click  Ok and close all power options.

Screenshorts

Windows Universal or Universal Windows – Part 2 #uwp #wpdev

Great minds think alike… its unlikely that you read the Part 1 its a bit TL;DR; to be frank.

Assuming you have a Windows Universal project or are starting there and are also interested in targeting Windows 10. Here is me creating a blank Windows 8.1 Universal project using Blank template.

This is what Windows Universal Project looks like… You have the Windows head (that’s what they called it), the Windows Phone head and the Shared head.

Whatever is in the shared head is copied to Windows & WindowsPhone head as the first step of compiling the project. Its a neater version of sharing a file by adding a file reference (Add existing file > Add link).

All shared stuff goes in Shared head. Anything specific to either head goes in Windows or WindowsPhone heads. Shared head usually includes all the Models, ViewModels, Services, Helpers, Behaviours and whatever stuff you don’t what to put in PCL (not everyone uses PCLs)

Lets add Windows 10 project to the solution

We need a few minor adjustments before Windows 10 project becomes a part of Windows Universal and thinks the same way. Lets delete the App.xaml and App.xaml.cs to start with.

Next right click unload Windows head and then right click and choose Edit

Scroll all the way down until you see something like this

Copy this line.

Now unload and edit the Windows 10 project. Scroll to the bottom to the exact same spot and paste the line without any changes.

Save and close. Reload both Windows head and Windows 10 project and that’s it. The Windows 10 universal project is now a part of Windows Universal project. The circle is complete.

All items in shared head are now a part of Windows 10 head and all you need to do is add Windows 10 head specific bits like you have for Windows 8.1 and Windows Phone 8.1

You build for store like you build before.

You have appx / appxbundle for

Windows (x86 / x64 / ARM)

WindowsPhone (ARM)

Windows 10 (x86 / x64 / ARM) compiled to Native