Word list by frequency based on Open Subtitle corpus 2018

A while ago I started working on newer version of Word list based 2018 OpenSubtitle Corpus. Downloading the data takes a while and can be frustrating – Microsoft Edge not being able to continue down in the background (should you close the main window) etc. I got distracted into writing a download manager.

Thanks To Hugo Lopez‘s comments on GitHub I was reminded again and I got started the 2nd time.

Based on comments from Hugo and others I added Language DetectionLanguage Detection to the list generator and generated the word lists.

I have not sanitised the lists in any way.
50k wordlists were only created only if overall word count exceeded 50k
Created ignored file that lists words identified as invalid. It is very likely that many words inside the ignored files are valid. Their frequency within the corpus is preserved should you wish to correct them.
With ja and fr a few input files contained invalid words. I added check to prevent the generator from crashing. Further processing of that file was not done.
From a given directory, only 1 file was processed. Any additional files in the input data were ingored.
The generated dataset can be found on Github along with code.

https://github.com/hermitdave/FrequencyWords
https://github.com/hermitdave/FrequencyWords/tree/master/content/2018

Be moar reactive with #uwpdev ?

This is the 4th and hopefully not the last post ever about Reactive Extensions with #uwpdev. After my 1st post, Jamie Mutton mentioned that writing async task code inside a subscription can lead to side effects and he posted a gist on various options.

I decided to try and reduce some of the async task code from subscription and into another observable following the first option Jamie suggested. Here’s what I had before.

The subscription in the constructor of the ViewModel.

this.ChannelDataService.ActivateChannelSubject
    .ObserveOn(RxApp.TaskpoolScheduler)
    .SubscribeOn(RxApp.TaskpoolScheduler)
    .Subscribe(async tuple =>
{
    if (tuple != null)
    {
        await this.ProcessChannelData();
    }
});

this.ChannelDataService.ChannelDataSubject
    .ObserveOn(RxApp.TaskpoolScheduler)
    .SubscribeOn(RxApp.TaskpoolScheduler)
    .Subscribe(async tuple =>
{
    if (tuple != null)
    {
        await this.ProcessChannelData();
    }
});

The View Model methods

private async Task ProcessChannelData()
{
    try
    {
        await semaphore.WaitAsync();

        await DispatcherHelper.ExecuteOnUIThreadAsync(() => this.IsBusy = true);

        var channelData = this.ChannelDataService.GetChannelData();
        await this.RenderChannelData(channelData);
    }
    finally
    {
        semaphore.Release();

        await DispatcherHelper.ExecuteOnUIThreadAsync(() => this.IsBusy = false);
    }
}

private async Task RenderChannelData(ChannelData channelData)
{
    if (channelData == null)
    {
        await DispatcherHelper.ExecuteOnUIThreadAsync(() => this.ResetData());

        return;
    }

    var configSettings = this.ChannelDataService.ConfigurationSettings;

    var itemData = await channelData?.ProcessItems(ChannelService.ChannelDictionary, ChannelService.ChannelList[0], this.AppSettings.Country);

    if (itemData == null)
    {
        return;
    }

    var puffData = channelData.ProcessPuffs(ChannelService.ChannelDictionary, ChannelService.ChannelList[0]);

    System.Diagnostics.Debug.WriteLine($"Item count is {itemData.Item1.Count}, Group count is {itemData.Item2.Count}");

    await DispatcherHelper.ExecuteOnUIThreadAsync(() =>
    {
        try
        {
            ViewHelper.SetStatusBar(this.Channel.StatusColor, Colors.White, this.AppSettings.ElementTheme);

            if (this.Items.Count > 0)
            {
                this.Items.Clear();
            }

            this.Items.AddRange(itemData.Item1);

            if (this.Content.Count > 0)
            {
                this.Content.Clear();
            }

            this.Content.AddRange(itemData.Item2);
        }
        catch { }

        try
        {
            if (PuffContent.Count > 0)
            {
                this.PuffContent.Clear();
            }

            this.PuffContent.AddRange(puffData);
        }
        catch { }
    });
}

I tried to split the code into two parts – Fetch and prepare data in one and render in another.

private async Task<ChannelData> ProcessChannelDataAsync()
{
    ChannelData channelData = null;
    try
    {
        await semaphore.WaitAsync();

        await DispatcherHelper.ExecuteOnUIThreadAsync(() =>
        {
            this.IsBusy = true;
        });

        await Task.Run(() =>
        {
            channelData = this.ChannelDataService.GetChannelData();

            if (channelData != null)
            {
                channelData.ProcessChannelData(ChannelService.ChannelDictionary, ChannelService.ChannelList[0], this.AppSettings.Country);
            }
        });
    }
    catch (JsonReaderException jsonEx)
    {
        Exception handledException = new Exception("Handled data parse exception", jsonEx);
        HockeyClient.Current.TrackException(handledException);
    }
    catch (SqliteException ex)
    {
        Exception handledEx = new Exception("Handled SQLite error getting channel data", ex);
        HockeyClient.Current.TrackException(handledEx);

        DialogService ds = new DialogService();
        await ds.ShowMessage("Application encountered serious error. Please reinstall the application.", SystemInformation.ApplicationName);
    }
    finally
    {
        semaphore.Release();
    }

    return channelData;
}

private async Task RenderChannelDataAsync(ChannelData processedData)
{
    try
    {
        await semaphore.WaitAsync();

        await DispatcherHelper.ExecuteOnUIThreadAsync(() =>
        {
            ViewHelper.SetStatusBar(this.Channel.StatusColor, Colors.White, this.AppSettings.ElementTheme);

            if (processedData == null)
            {
                this.ResetData();

                return;
            }

            if (this.Items.Count > 0)
            {
                this.Items.Clear();
            }

            if (processedData != null)
            {
                this.Items.AddRange(processedData.ProcessedItems);
            }

            if (this.Content.Count > 0)
            {
                this.Content.Clear();
            }

            if (processedData != null)
            {
                this.Content.AddRange(processedData.GroupedItems);
            }

            if (PuffContent.Count > 0)
            {
                this.PuffContent.Clear();
            }

            if (processedData != null)
            {
                this.PuffContent.AddRange(processedData.PuffGroupedItems);
            }

            this.IsBusy = false;
        });
    }
    catch { }
    finally
    {
        semaphore.Release();
    }
}

Now I could chain the original observation with a new observable(s) created out of async method(s) as suggested by Jamie Mutton in his gist.

The extension method to convert async method to an observable is simply ToObservable().

this.ChannelDataService.ActivateChannelSubject
    .Where(t => t != null)
    .Select(_ => Observable.Defer(() => ProcessChannelDataAsync().ToObservable()))
    .Concat()
    .Select(processedData => Observable.Defer(() => this.RenderChannelDataAsync(processedData).ToObservable()))
    .Concat()
    .Subscribe(_ =>
    {
        System.Diagnostics.Debug.WriteLine($"Active channel changed");
    });

this.ChannelDataService.ChannelDataSubject
    .Where(t => t != null)
    .Select(_ => Observable.Defer(() => ProcessChannelDataAsync().ToObservable()))
    .Concat()
    .Select(processedData => Observable.Defer(() => this.RenderChannelDataAsync(processedData).ToObservable()))
    .Concat()
    .Subscribe(processedData =>
    {
        System.Diagnostics.Debug.WriteLine($"Channel data updated");
    });

Performance isn’t much significantly different from earlier but the code is now more reactive than before. The subscription as I was told is as light as can be and there is little room for side-effects

Happy coding.

Reactive Extensions with #uwpdev Part 3

The way I understood Rx, it was designed to do away with how we (developers) handle events.

A ListView control provides a few ways of handling how developers deal with user interaction. You can have ItemClick and disable Item Selection. You could have Item Selection with ability to select more than one Item.
My personal favourite is to allow ItemClick and to bind a Command property in ListViewExtensions in UWP Toolkit.

There are times when you really need to handle the event in code behind – I know purists would have a day here but say you had a video previews running in a list view. You don’t want all of them running all the time – you want to ensure that only those rendered on screen are running the video.
How would one do that ?

The usual route would be to find the ScrollViewer in the ListView control and hook into the ViewChanged event.
Every time ViewChanged event is raised, OnScrollViewerViewChanged method will be called.

By every time I mean every time.. could be maybe times every second. You then start have to add hacks like lastChecked time and if Time Elapsed is greater than 100ms etc etc.

private void WireUpListViewScroll(ListView listView)
{
    if (listView == null)
    {
        return;
    }

    if (scrollViewer != null)
    {
        scrollViewer.ViewChanged -= OnScrollViewerViewChanged;
    }

    var scrollViewer = listView.FindDescendant<ScrollViewer>();

    if (scrollViewer == null)
    {
        return;
    }

    scrollViewer.ViewChanged += OnScrollViewerViewChanged;
    
    ProcessChannelUserControls(listView, true);
}

private void OnScrollViewerViewChanged(object sender, ScrollViewerViewChangedEventArgs e)
{
    this.ProcessChannelUserControls(listView);
}

So lets look up what Reactive extensions has for us. For starters, Observable.FromEventPattern allows you to create an observable from event and to subscribe to it. No need to unsubscribe from event handers etc.. just dispose the subscription.

Notice I also use Throttle with a TimeSpan.. I don’t want too many hits – once every 0.35 seconds seemed sufficiently responsive for my use and what’s what I used. the scrollViewerSub I use is a SerialDisposable – every time I add a new subscription, it disposes the old one.

private void WireUpListViewScroll(ListView listView)
{
    if (listView == null)
    {
        return;
    }

    var scrollViewer = listView.FindDescendant<ScrollViewer>();

    if (scrollViewer == null)
    {
        return;
    }

    this.scrollViewerSub.Disposable = Observable.FromEventPattern(scrollViewer, "ViewChanged")
        .Throttle(TimeSpan.FromSeconds(0.35))
        .ObserveOn(CoreDispatcherScheduler.Current)
        .SubscribeOn(TaskPoolScheduler.Default)
        .Subscribe(x =>
        {
            this.ProcessChannelUserControls(listView);
        });

    ProcessChannelUserControls(listView, true);
}

Now that the ViewChanged logic is in place, let’s look at the rest of the code for toggling video previews.

private async void ProcessChannelUserControls(ListView listview, bool initial = false)
{
    try
    {
        var isp = listview.ItemsPanelRoot as ItemsStackPanel;

        if (isp == null)
        {
            return;
        }

        await ToggleAnimatedPreviews(listview, isp, initial);
    }
    catch { }
}

private async Task ToggleAnimatedPreviews(ListView listView, ItemsStackPanel isp, bool initial)
{
    var firstGroupPos = isp.FirstVisibleIndex;
    var lastGroupPos = isp.LastVisibleIndex;

    if (firstGroupPos == -1 && lastGroupPos == -1)
    {
        firstGroupPos = 0;
        lastGroupPos = 0;
    }

    if (initial)
    {
        DependencyObject obj = null;
        while (true)
        {
            obj = listView.ContainerFromIndex(0);

            if (obj != null)
            {
                break;
            }

            await Task.Delay(100);
        }
    }

    for (int i = 0; i < listView.Items.Count; i++)
    {
        var container = listView.ContainerFromIndex(i);

        var vp = container?.FindDescendant<AnimatedPreviewPlayerUC>();

        if (i >= firstGroupPos && i <= lastGroupPos)
        {
            vp?.Play();
        }
        else
        {
            vp?.Pause();
        }
    }
}

These are some of the places I have used Reactive Extensions whilst doing #uwpdev with MVVM. I am still learning.

Reactive Extensions with #uwpdev Part 2

In my post yesterday I talked about adding a few reactive bits inside MVVM using ReactiveUI.

Jamie Mutton mentioned that writing async task code inside a subscription can lead to side effects and he posted a gist on various options. I haven’t tried those yet – I will do so soon. I have mentioned something about not being a purist and I found it easier to inject Rx elements into MVVM whilst following known (to myself) programming style.
Being a slow learner, I prefer to inject things slowly. Add reactive bits at your own pace. You can go the whole 9 yards or go a step at a time.

Today let’s see how to add Reactive Extensions to Service layer. MVVM should really be MVVMS but its a mouthful and not a palindrome.

The Daily Mail Online app has many channels that group content as desired by the editors. Within the app, there’s a Channel list and the content is displayed within FlipView.

Channel list is bound to a RelayCommand (MVVMLight) – setting the correct ChannelIndex. The FlipView is bound to a list of ChannelViewModels with ChannelIndex as the SelectedIndex. When the channel changes, the Data Service is notified and new data is requested.

The sequence is as follows

  1. When Channel Index changes, it pushes a new Tuple through the Active Channel Subject.
  2. When Active Channel Subject publishes a new item:
    • the subscriber in the service, if data is out of data, it downloads new data set and saves it to database, publishing the tuple through to Channel Data Subject
    • the subscriber in the view model, pulls and renders data from database
  3. When Channel Data Subject publishes a new item, the view model pulls it from database and renders it
public int ChannelIndex
{
    get
    {
        return this.channelIndex;
    }

    set
    {
        try
        {
            this.RaiseAndSetIfChanged(ref this.channelIndex, value);
            this.RaisePropertyChanged(nameof(this.Channel));
            this.RaisePropertyChanged(nameof(this.ChannelViewModel));

        }
        catch { }

        var channel = this.Channel;

        SubChannel selectedSub = null;

        if (channel.HasSubChannels)
        {
            selectedSub = channel.GetDefaultSubChannel();
        }

        this.ChannelDataService.SetActiveSource(new Tuple(channel, selectedSub));
    }
}

The Channel Data Service is a monstrosity in itself but that’s another issue. What I have in the service are Reactive Subject instances to publish – in this instance a Tuple for both ActiveChannel and when a new data set is available.

Subjects are considered to be educational material – however in absence of any alternative, I decided to start with those.

public BehaviorSubject ActivateChannelSubject { get; private set; }

public BehaviorSubject ChannelDataSubject { get; private set; }

As you saw above, the ChannelIndex calls a method SetActiveSource. This merely pushes item to the Subject.

public void SetActiveSource(Tuple tuple)
{
    this.SelectedChannelTuple = tuple;

    this.ActivateChannelSubject.OnNext(tuple);
}

Within the service itself, I have a subscription to this subject. Other subscriber includes the MainViewModel for the app

this.ActivateChannelSubject.Subscribe(async tuple =>
{
    try
    {
        // checks to see if existing data is up to date.
        // If it isn't, it pulls necessary content
        await this.ProcessChannel(tuple, true);
    }
    catch { }
});
public async Task ProcessChannel(
    Tuple channelTuple,
    bool notifySubscribers,
    bool forceRefresh = false)
{
    if (!NetworkHelper.Instance.ConnectionInformation.IsInternetAvailable)
    {
        return;
    }

    try
    {
        // check if data is up to date

        // download content as json

        // inject new data into the local db

        if (notifySubscribers)
        {
            this.ChannelDataSubject.OnNext(channelTuple);
        }
    }
}

I remember the warning Jamie gave about the side effects when using async task in subscription. I promise to take a look at the alternative but this is what it is right now..
The try catch around ProcessChannel are just to make sure any exceptions do not take the app down as async lambda is async void and it can burn the house down if not very careful.

Lets took at the subscriptions in the view model’s constructor

this.ChannelDataService.ActivateChannelSubject
    .ObserveOn(RxApp.TaskpoolScheduler)
    .SubscribeOn(RxApp.TaskpoolScheduler)
    .Subscribe(async tuple =>
{
    if (tuple != null)
    {
        await this.ProcessChannelData();
    }
});

this.ChannelDataService.ChannelDataSubject
    .ObserveOn(RxApp.TaskpoolScheduler)
    .SubscribeOn(RxApp.TaskpoolScheduler)
    .Subscribe(async tuple =>
{
    if (tuple != null)
    {
        await this.ProcessChannelData();
    }
});

The View Model itself pulls the data, groups it and renders it on UI thread

private async Task ProcessChannelData()
{
    try
    {
        await semaphore.WaitAsync();

        await DispatcherHelper.ExecuteOnUIThreadAsync(() => this.IsBusy = true);

        var channelData = this.ChannelDataService.GetChannelData();
        await this.RenderChannelData(channelData);
    }
    finally
    {
        semaphore.Release();

        await DispatcherHelper.ExecuteOnUIThreadAsync(() => this.IsBusy = false);
    }
}

private async Task RenderChannelData(ChannelData channelData)
{
    if (channelData == null)
    {
        await DispatcherHelper.ExecuteOnUIThreadAsync(() => this.ResetData());

        return;
    }

    var configSettings = this.ChannelDataService.ConfigurationSettings;

    var itemData = await channelData?.ProcessItems(ChannelService.ChannelDictionary, ChannelService.ChannelList[0], this.AppSettings.Country);

    if (itemData == null)
    {
        return;
    }

    var puffData = channelData.ProcessPuffs(ChannelService.ChannelDictionary, ChannelService.ChannelList[0]);

    System.Diagnostics.Debug.WriteLine($"Item count is {itemData.Item1.Count}, Group count is {itemData.Item2.Count}");

    await DispatcherHelper.ExecuteOnUIThreadAsync(() =>
    {
        try
        {
            ViewHelper.SetStatusBar(this.Channel.StatusColor, Colors.White, this.AppSettings.ElementTheme);

            if (this.Items.Count > 0)
            {
                this.Items.Clear();
            }

            this.Items.AddRange(itemData.Item1);

            if (this.Content.Count > 0)
            {
                this.Content.Clear();
            }

            this.Content.AddRange(itemData.Item2);
        }
        catch { }

        try
        {
            if (PuffContent.Count > 0)
            {
                this.PuffContent.Clear();
            }

            this.PuffContent.AddRange(puffData);
        }
        catch { }
    });
}

This I believe is like 11/2 steps at max.. There a long way to go.. much to refine but things work. I force use of TaskPool unless I need a UI.. in that case, queue it up on the Dispatcher.

I have done it the non reactive way before and whilst it all works, it feels better with reactive model. Things split into independently observables

Reactive Extensions with #uwpdev

Traditionally modern Windows application development (something done in the last 10 years) would involve using XAML. XAML is a great way to define a UI – it addition to it being declarative, it also supports Binding and Commanding.
MVVM – Model View View-Model has become the defacto pattern when developing XAML based applications… even a slow learner like myself eventually graduated to MVVM.

Anything that uses Binding needs to implement INotifyPropertyChanged interface by means on invoking PropertyChanged event to notify the UI of change.

MvvmLight toolkit which provides a large collection of helpful plumbing (RelayCommand, SimpleIoC, NavigationService and more) also provides an object called ObservableObject. This implements the INotifyPropertyChanged and devs just have to inherit from it and introduce their own properties ready for binding.
ReactiveUI a Reactive Extensions MVVM toolkit provides a similar class called ReactiveObject.

namespace ReactiveUI
{
    //
    // Summary:
    //     ReactiveObject is the base object for ViewModel classes, and it implements INotifyPropertyChanged.
    //     In addition, ReactiveObject provides Changing and Changed Observables to monitor
    //     object changes.
    [DataContract]
    public class ReactiveObject : IReactiveNotifyPropertyChanged, IHandleObservableErrors, IReactiveObject, INotifyPropertyChanged, INotifyPropertyChanging, IEnableLogger
    {
        protected ReactiveObject();

        //
        // Summary:
        //     Represents an Observable that fires *before* a property is about to be changed.
        [IgnoreDataMember]
        public IObservable Changing { get; }
        //
        // Summary:
        //     Represents an Observable that fires *after* a property has changed.
        [IgnoreDataMember]
        public IObservable Changed { get; }
        //
        [IgnoreDataMember]
        public IObservable ThrownExceptions { get; }

        public event PropertyChangingEventHandler PropertyChanging;
        public event PropertyChangedEventHandler PropertyChanged;

        //
        public bool AreChangeNotificationsEnabled();
        public IDisposable DelayChangeNotifications();
        //
        public IDisposable SuppressChangeNotifications();
    }
}

When creating observable objects, even ViewModel I tend to derive from ReactiveObject. Any derived class has access to tons of ReactiveExtensions.

For example, I mentioned that my work app Daily Mail Online is a SPA UWP app. I use a property called ViewMode to identify what View is loaded / shown to the user and what isn’t.

public ViewModes ViewMode
{
    get
    {
        return this.viewMode;
    }

    private set
    {
        this.RaiseAndSetIfChanged(ref this.viewMode, value);
    }
}

The method RaiseAndSetIfChanged is another extension method available to a reactive object. This raises PropertyChanged event if value has changed and UI needs to be informed. This would be great if ViewMode was bound to XAML but I wanted to observe any changes from different places. What I created was an observable using another extension method

public IObservable ViewModeObservable { get; private set; }
this.ViewModeObservable = this.WhenAnyValue(x => x.ViewMode)
    .Where(x => x != ViewModes.NotSet)
    .ObserveOn(CoreDispatcherScheduler.Current)
    .SubscribeOn(TaskPoolScheduler.Default);

I used an extension method called WhenAnyValue. It allows one to convert any property the raises PropertyChanged event as Observable.

Notice that I have set ObserveOn and SubscribeOn methods – those are not necessary here rather a subscriber can choose to set those. I set those here because I know this needs to be run on UI thread and any subscribers would need to do the same.
Now lets see how we can observe changes raised by this observable.

I hook into the ViewModel’s observable property in the View’s code behind. I am not an MVVM purist and I am happy to get into code behind if needed

this.MainViewModel.ViewModeObservable.Subscribe(async viewMode =>
{
    await _semaphoreViewMode.WaitAsync();

    try
    {
        switch (viewMode)
        {
            case ViewModes.Article:

                // do something
                break;

            case ViewModes.RelatedArticle:

                // do something
                break;

            case ViewModes.Channel:

                // do something
                break;

            case ViewModes.ReadLater:

                // do something
                break;

            case ViewModes.Comments:

                // do something
                break;

            case ViewModes.Search:

                // do something
                break;

            case ViewModes.Topic:

                // do something
                break;

            case ViewModes.Profile:

                // do something
                break;
        }
    }
    finally
    {
        _semaphoreViewMode.Release();
    }
});

The Subscribe extension method returns an instance of IDisposable. One can use that or use SerialDisposable to auto dispose previous subscription when a new subscription happens.

One can Subscribe to an Observable as many places as one needs.

Note: Its very likely that I am doing Reactive Extensions all wrong. Feel free to suggest better ways of using Rx

Single Page Application with #uwpdev

Over the next few days I will push a set of blog posts detailing the design of Daily Mail Online app for Windows. This has been due for at least 12 months now.

I am not going to discuss the Content published by the App rather the App in itself – I am the creator of the App and have no control over the content.

Traditionally most applications have multiple views / pages. User is presented with different pages depending upon the data to be shown. Page transition theme makes navigation between pages a joy.

I had issues with page navigation – especially when the page you are navigating to contains a FlipView control and the item to be displayed is anything but the first item – there is an obvious rendering issue where the first item renders momentarily before the correct item is shown – really makes me want to kick someone in the nuts every time I see it.

During my rework of Daily Mail Online app for desktop last year, I decided to go the SPA route. SPA stands for Single Page Application. Whilst it has been around for a very long time, I first heard it from Benjamin Howarth when he did a talk at London Mobile Dev user group back in 2015

So the idea of SPA is simple
* Get all necessary data
* Display correct view depending upon what the user desires.
* Maintain pseudo navigation to maintain sanity.

Daily Mail Online app has the following views:
Channel view
Article view
Related Article View
Comment View
Topic View
Search View
Profile View
Read Later View
Photo Gallery View
Video View

In early days of UWP, one could use Deferred Loading but there was no way to unload anything you didn’t want. That meant that from a Visual Tree point of view it was messed up – all these views – all rendered and in memory just hidden away when not needed.

Last year with Creators update, Windows Dev team introduced a feature called x:Load. It allows dynamically loading and unloading of controls – providing an experience similar to visibility toggle while having a lighter footprint.

<ListView
    x:Name="ArticleListView"
    Grid.Row="1"
    Grid.RowSpan="2"
    x:Load="{x:Bind MainViewModel.LoadSingleArticleView, Mode=OneWay}"
    Background="{ThemeResource ContentBackgroundBrush}"
    BorderBrush="{ThemeResource ContentBackgroundBrush}"
    BorderThickness="0"
    ItemContainerStyle="{StaticResource GroupItemStyle}"
    ItemTemplateSelector="{StaticResource ArticleLayoutSelector}"
    ItemsSource="{Binding ElementName=mainPage, Path=DataContext.RelatedItemContent, Mode=OneWay}"
    Loaded="OnArticleListViewLoaded"
    SelectionMode="None"
    ShowsScrollingPlaceholders="False"
    VirtualizingStackPanel.VirtualizationMode="Recycling" />

The only requirement is that you need to Name the control and Laoding / Unloading can be done by binding to VM property – it can’t really get any simpler.

Along with SPA design, I use ReactiveUI along with traditional MVVM within the Daily Mail Online App. More on injecting ReativeX goodness within MVVM app in next post.

End of Cineworld app for #Windows and issues facing indie dev #uwpdev

Back in the summer of 2016, I sat waiting for a movie to start at Cineworld. Wanted to check something, I was forced to use the mobile website. At that point I wondered if they had an API and within 2 days I had a version of the app ready for Windows Phone store.

A few months down the line, I ported it to Windows 8 and then ended up using Azure to generate dataset for the apps.  For the last 2 years or so I have been paying for the service.. it wasn’t significant but still odd 200 a month and having looked at IAP purchase in the last 2 months I decided that it was time to call it a day.

While Microsoft might say mobile didn’t work for them, I’d beg to differ. Windows Mobile version of the app had nearly 4k downloads (3976) vs desktop app with 256 downloads. YES a full 256 downloads.

Mobile app – Windows Phone app (WP8x and Windows 10 Mobile)

Cineworld-Mobile

Desktop app – Windows 8 app (Windows 8x and Windows 10 Desktop)

Cineworld-Desktop

That tells you all you need to know about consumption of certain apps on desktop vs mobile – it might not be global but it definitely is so in UK.

Now lets look at IAP.

IAP for Mobile app

Cineworld-IAP-Mobile

IAP for Desktop app

Cineworld-IAP-Desktop

Above are for the last 2 months – minus governmental and store taxation, I could have paid for the Azure service for a month.

Having decided that it wasn’t economical, the fact that Microsoft officially acknowledged end of the road for mobile was sufficient to determine that the app will never ever pay for itself.

The source for the Apps and Azure services is available on GitHub

Most code was before I adopted MVVM for Windows Development so please except my apologies for whatever you find in there.

Happy coding

Finding system fonts for use with XAML UWP app #uwpdev

Back in 2011 in my early days of Windows Phone 7 development, I created Alarm Clock app. Over a period of time, I upgraded it to SL8x and made it available on Windows 8x devices. It has seen many iterations and it made sense to rework it for UWP so I don’t have 2 distinct apps any longer – just one that scales across devices.

What was unique about this app ? It has some custom font and alarm sounds.. weird and wonderful – lets just say unusual.

This time around I thought that maybe just maybe I should also allow user to select system fonts!. So how does one go about it ? Well SO #FTW 😛 Filip Skakun replied on http://Loading list of available fonts using c# in winrt mentioned DirectWrite and linked to a git repos

https://github.com/christophwille/winrt-snippets/blob/master/EnumerateFonts/EnumerateFonts/InstalledFont.cs
and
https://github.com/sharpdx/SharpDX-Samples/blob/master/Desktop/DirectWrite/FontEnumeration/Program.cs

For the sake of showing time, I wasn’t interested in symbol fonts otherwise the source is identical.

private void LoadInstalledFonts()
{
    var factory = new SharpDX.DirectWrite.Factory();
    var fontCollection = factory.GetSystemFontCollection(false);
    var familyCount = fontCollection.FontFamilyCount;
            
    for (int i = 0; i < familyCount; i++)
    {
        var fontFamily = fontCollection.GetFontFamily(i);
                
        var familyNames = fontFamily.FamilyNames;
        int index;

        if (!familyNames.FindLocaleName(CultureInfo.CurrentCulture.Name, out index))
            familyNames.FindLocaleName("en-us", out index);

        string name = familyNames.GetString(index);

        using (var font = fontFamily.GetFont(index))
        {
            if (font.IsSymbolFont)
            {
                continue;
            }
        }

        this.AvailableFonts.Add(new FontFamily(name));
    }
}

The DirectWrite API is exposed by SharpDX and you need to add a reference to SharpDX.Direct2D1 using Nuget Package Manager

That is all that is required. Now the system fonts are available for use however you please. I myself allow user to chose what font they want to use.

Happy Coding

Operating system analysis of a small #uwp app user base #uwpdev

In 2015, I released Windows Phone 8.1 and Windows 10 version of the Daily Mail Online app. The apps uses tracking to monitor usage etc. In the last set of builds for the first iteration of my work there, I set the OS to be Windows Phone 8.1, Windows 10 and Windows 10 Mobile. Whilst it was sufficient back then, it meant that I did not have sufficient data to set the Min Target SDK the 2nd time around.

Let me start by saying that my data is mostly UK centric.

Such data hasn’t so far been released by Microsoft and to ensure that I targeted as many users as possible I settled for a min target SDK to be 10240 (the first version of Windows 10).

I still have users on the early version of the app and those will never upgrade. There are significant users on Windows Phone 8.1 (dropping at a steady rate)

I have been monitoring this numbers every day and I finally decided to modify the Min SDK to 14939. This is why.

Of the Windows 10 users, majority were on 14393. 0.7% users are still on 10240, around 1.9% are on 10586.

I found 16 different builds of 10240 in use amongst those 0.7%

21 different builds for 10586

With number of Insiders matching those on older generations, its time move forward. Min target for next update is 14393 and a total of 33 Windows 10 builds to support.

Using Composition Animations in your #uwp app

As we speak a new version of Daily Mail Online is being published to store. I am not going to discuss the content rather stick with code.

I have been testing staged rollout for a week now and that gave me valuable insights. I got errors I could not have reproduced myself. However all this time I was using UWPCommunityToolkit animations – Fade and Offset. Testing out nightly builds post 1.2 release I noticed that the app start time would significantly spiral with initial load taking over 20 seconds. The toolkit animations whilst providing a simple API on the surface are very extensive underneath and talking to folks I decided to create a custom set.

I have in snippet below two border controls

Border b = item.FindDescendantByName("ChannelBackground") as Border;
Border separator = item.FindDescendantByName("ChannelSeparator") as Border;

using toolkit extensions I did

b.Offset(offsetY: -65, duration: animationduration).Start();
separator.Fade(1, animationduration).Start();

so after a big round of search and copy pasting code from around the net, I finally had my own set. To make the change easier, I decided to use different method names so I could test individual bits without affecting it all.

b.AnimateOffsetY(-65, animationduration);
separator.AnimateOpacity(1, animationduration);

One thing that caught me off guard. To not show separator when XAML loads, I set Opacity to 0. With opacity to set, Composition API could not change its opacity any longer. As a fallback I use Storyboard and they seem to work just fine. Took me a while to figure out so now you will notice that in animation extensions, I check UIElement opacity and set it to 1 if it is zero.

Here is the animation extensions.

public static class AnimationExtensions
{
    public static bool UseCompositionAnimations => ApiInformation.IsApiContractPresent("Windows.Foundation.UniversalApiContract", 2); // SDK >= 10586

    public static void AnimateOpacity(this UIElement element, float value, double duration = 150)
    {
        if (UseCompositionAnimations)
        {
            if (element.Opacity == 0) // composition animations doesn't have any effect if it is zero
            {
                element.Opacity = 1;
            }
                
            var xamlVisual = ElementCompositionPreview.GetElementVisual(element);

            if (duration <= 0)
            {
                xamlVisual.Opacity = value;
                return;
            }

            var compositor = xamlVisual.Compositor;

            var animation = compositor.CreateScalarKeyFrameAnimation();
                
            animation.Duration = TimeSpan.FromMilliseconds(duration);
            animation.DelayTime = TimeSpan.Zero;
            animation.InsertKeyFrame(1f, value);

            xamlVisual.StartAnimation("Opacity", animation);
        }
        else
        {
            if (duration <= 0)
            {
                element.Opacity = value;
                return;
            }

            AnimateDoubleProperty(element, "Opacity", value, duration);
        }
    }

    public static void AnimateOffsetY(this UIElement element, float value, double duration = 150)
    {
        if (UseCompositionAnimations)
        {
            var xamlVisual = ElementCompositionPreview.GetElementVisual(element);

            var offsetVector = new Vector3(0, value, 0);

            if (duration <= 0)
            {
                xamlVisual.Offset = offsetVector;
                return;
            }

            var compositor = xamlVisual.Compositor;

            var animation = compositor.CreateVector3KeyFrameAnimation();

            animation.Duration = TimeSpan.FromMilliseconds(duration);
            animation.DelayTime = TimeSpan.Zero;
            animation.InsertKeyFrame(1f, offsetVector);

            xamlVisual.StartAnimation("Offset", animation);
        }
        else
        {
            var transform = GetAttachedCompositeTransform(element);

            if (duration <= 0)
            {
                transform.TranslateY = value;
                return;
            }

            string path = GetAnimationPath(transform, element, "TranslateY");

            AnimateDoubleProperty(element, path, value, duration);
        }
    }

    private static string GetAnimationPath(CompositeTransform transform, UIElement element, string property)
    {
        if (element.RenderTransform == transform)
        {
            return $"(UIElement.RenderTransform).(CompositeTransform.{property})";
        }

        var group = element.RenderTransform as TransformGroup;

        if (group == null)
        {
            return string.Empty;
        }

        for (var index = 0; index < group.Children.Count; index++)
        {
            if (group.Children[index] == transform)
            {
                return $"(UIElement.RenderTransform).(TransformGroup.Children)[{index}].(CompositeTransform.{property})";
            }
        }

        return string.Empty;
    }

    private static CompositeTransform GetAttachedCompositeTransform(UIElement element)
    {
        CompositeTransform compositeTransform = null;

        if (element.RenderTransform != null)
        {
            compositeTransform = element.RenderTransform as CompositeTransform;
        }

        if (compositeTransform == null)
        {
            compositeTransform = new CompositeTransform();
            element.RenderTransform = compositeTransform;
        }

        return compositeTransform;
    }

    private static Storyboard AnimateDoubleProperty(
        this DependencyObject target,
        string property,
        double to,
        double duration = 250,
        EasingFunctionBase easingFunction = null)
    {

        var storyboard = new Storyboard();
        var animation = new DoubleAnimation
        {
            To = to,
            Duration = TimeSpan.FromMilliseconds(duration),
            EasingFunction = easingFunction ?? new SineEase(),
            FillBehavior = FillBehavior.HoldEnd,
            EnableDependentAnimation = true
        };

        Storyboard.SetTarget(animation, target);
        Storyboard.SetTargetProperty(animation, property);

        storyboard.Children.Add(animation);
        storyboard.FillBehavior = FillBehavior.HoldEnd;
        storyboard.Begin();

        return storyboard;
    }
}

These are very lightweight animations. If you encounter weird slow downs, maybe consider rolling out your own animation extensions like above. Happy coding