Open letter to Mackenzie Scott: 5 Why’s with Effective Altruism – Invest in Post Scarcity Solutions

This is an open letter to MacKenzie Scott and anyone else donating money to charities.

Dear MacKenzie Scott,

I am impressed with you signing the Giving Pledge and the sheer amount of money you’ve been giving. The world collectively thanks you.

However, when looking at how you’ve been selecting and aiming your donations I feel there’s some aspects missing to the approach and I think a 5 Why’s analysis would work well.

The 5 Why’s is an approach to working out the root cause of a defect or problem by repeating the question “Why?”. Each answer forms the basis of the next question.
You repeat the process until you come to the root cause (which can often be 5 times). You then apply corrections and fixes at each of the levels.

It’s that last point which I’ve found many charities, groups and organisations are unable to do as they exist only work on a certain level.

An example 5 Why’s:

 The vehicle will not start.

  1. Why? – The battery is empty.
  2. Why? – The alternator is not functioning as the alternator belt has broken.
  3. Why? – The alternator belt was well beyond its useful life, but not replaced.
  4. Why? – The vehicle has not been maintained.
  5. Why? – The vehicle is so old that maintenance costs are too high.

There’s a few different funding strategies you can apply. Broadly speaking I describe them as the pyramid, upside down pyramid, cylinder and spotlight.

The Pyramid Funding scheme

The pyramid focuses most of the money and resources on the first Why and applies less money as you go up the pyramid. In the vehicle example you’d spend the most money on buying a new battery, some on an alternator belt and a little bit towards a service, but barely even browse your phone whilst on the toilet looking at new cars.

Inverted Pyramid funding scheme

The Inverted Pyramid is the opposite of the Pyramid funding scheme. In this case you might first get your car battery recharged and put on an alternator belt, but you’d spend most of your time and money on buying a new electric car. That way you don’t have to worry about all the other issues which will soon get worse and stop it from working in other ways.

The Cylinder / Tube funding scheme is straight forward. You just spend about the same amount of time and money on all the sections. So you might have the technician get the alternator belt changed straight away and the car jump started. You might also buy a car diagnostic system so you can investigate what other issues might arise and fend them off before they cause the car to break down on the freeway, blocking up traffic. You’d then buy an eBike as a backup for those days when your car is in the shop for repairs, which will be often enough you’ll also need a great raincoat for when riding the bike in the rain, plus get a lock for it and it’ll need it’s own servicing occasionally.

The Spotlight funding scheme is simple, you focus on a single why. Usually the first one which is the symptom. This is usually done without regard or investigation into the others. e.g You hear that the battery is dead and you get the battery replaced. You’ll soon find the new battery dead again and instead of recharging it or repairing the alternator belt you complain about being sold a faulty battery.

As per your post about 384 ways to help:

Some are filling basic needs: food banks, emergency relief funds, and support services for those most vulnerable. Others are addressing long-term systemic inequities that have been deepened by the crisis: debt relief, employment training, credit and financial services for under-resourced communities, education for historically marginalized and underserved people, civil rights advocacy groups, and legal defense funds that take on institutional discrimination.

384 ways to help

I can see you and the team are addressing some of the systemic issues.

However none attempt a system level redesign.

Thankfully with all the due diligence and work you’ve done in investigating the charities it seems that you aren’t using the spotlight funding scheme. You use the word “systems” 5 times between the two posts and the charities selected seem to have a mix of what I would consider the first 2 or 3 why’s with a Pyramid funding scheme. Most of the focus is on the short term problems with some addressing in-system change. But I couldn’t see any that are addressing the root cause of most of these issues.

Obviously not all problems have a single root cause. However many of the causes you were targeting, including Racial, LGBTQ+ and Gender Equity as well as Functional Democracy, Public Health, Climate Change and more are a result of failures of our current socio-economic system. In the car analogy, the focus has been on repairing the car instead of getting a new one and working out how to use it.

Your current focus is especially on those affected by Covid19 so a 5 whys analysis would look something like this:

Q1. Why are so many more people starving, homeless or simply struggling to meet basic needs?

A1. Because they need a job to survive but they’ve been laid off in staggering numbers.

Actions:

  • Directly give people money
  • Give people food and water
  • Give people shelter
  • Get people jobs
  • Transition to a UBI or UBS (although be careful)

Q2. Why have people been laid off?

A2. Due to the lockdowns and economic downturn because of the Covid19 pandemic.

Actions:

  • Stop or reduce the effects of the pandemic as much as possible
  • Help invest in vaccines (Bill and Melinda Gates foundation did this well)
  • Help reduce vaccine red-tape (this has been done)
  • Help reduce the misinformation around vaccines so enough people get vaccinated we’ll have herd immunity (there’s a whole other issue about algorithms prioritising screen time causing issues with our collective sense making here)

Q2. Why can’t the government or the markets save us from this situation?

A2. Because there’s political and economic issues to simply printing more money.

Actually the governments of the world have been printing lots of money but there’s core system issues, from increased inflation that will cause the poor to have even less money, to technological unemployment. All while because of the profit motive mixed with our values system we are creating more and more externalities causing grave environmental damage and human suffering.

Q5. Why aren’t we changing to a different system?
A5. We should be working towards transitioning to an abundance centered society (also known as post-scarcity), but there’s not nearly enough resources devoted to it.

There’s a number of groups working towards a post-scarcity society. They include The Zeitgeist Movement, The Venus Project, Game B, Social Ecology / Communalism and more.

I define a post-scarcity / abundance centered society as one where at least the necessities of life are free, for everyone on the planet.

There’s many aspects to a potential abundance centered society. Most require a full systems redesign using Cradle to Cradle materials flow so things are designed to last, or to be bio-nutritional (good for the environment). They’ll use access abundance, aim to have better collective sense and decision making. Obviously everything from automated production to the scientific method are all important.

For a better overview of both the issues of our current system and an overview of a new one I highly recommend Zeitgeist Moving Forward.

I’ve also got a presentation about transitioning to a post-scarcity society.

Cheers!

Michael Kubler

michael@zeitgeist-info.com

2020 – A year in review

Here’s a general look at my year of 2020.

The year that started off with epic bushfires in Australia, the Taal volcano in Philippines erupting and will be known as the year of Covid19. There was the massive explosion in Beirut, even brain eating amoebae in a part of the US water supply, the Arecibo observatory collapsed and a whole lot of political shenanigans in USA. Although at least Trump is on his way out, even if he’s flailing about as he goes. Thankfully the year hasn’t ended with an Alien invasion or something crazy.

This year. It’s been an interesting one. Certainly a start of the decade worth remembering.

For me, the main highlight of the year is that it’s when Xavier Jayden Cabahug Kubler was born, on the 15th of January. My first born Son and the start of me becoming a father and having a family with Jen.

It’s the year I declared my plan for helping transition to a Post-Scarcity society, with the next 5 years being prep and then 30+ years of transition.

It’s the year I started working on the Gather Together video analysis system that’s aimed at helping Stock Footage content creators keyword their content.

I also started but haven’t been able to work much on an app with my Sister for helping parents dealing with their kids medical needs (more to come later).

Unlike most people, the lockdowns barely affected my work. I have been working remotely for a few years now and it’s been working well. I’m still learning about myself and developing new coping mechanisms or better procedures.

Today, New Years Eve the 31st of Dec 2020 is the day when tomorrow is next year.

It’s been:

  • 933 days since Jen and I first met (2.6 years).
  • 676 days since I moved to the Philippines and started living with Jen. Also more than that since I’ve seen my family except via video call.
  • 351 days since Xavier was born (making him 50.1 weeks or 11.5 months old).


Family:

  1. The big obvious thing is that Xavier was born.
  2. I’m still living in Pasig, Metro Manila, Philippines with Jen, Xavier, our pug dog Maui, and our fish.
  3. It’s been amazing seeing Xavier grow up. He can now walk some steps unassisted and nearly says mumma and dadi. He’s very feisty and I hope he’ll grow up to be geisty.

Work:

  • I’m still doing web development for Mozzler. I’ve tracked at least 1,047 hours of work and I love working with Chris Were.
  • Worked on Viterra, Drakes, Drivible and some other projects. Mostly Yii2 with our own framework extras, but also some NodeJs. I’ve got 745 contributions listed on GitHub
Github Contributions
  • I started work on my own video analysis platform. The core works but I’m still not yet ready to release it. I’ve worked about 128 hrs on it this year, mostly some Sunday’s since July.
  • I have been working on stock footage creation in the mean time as well. Currently have made USD$503.73 in total from BlackBox submitted stock footage.
  • Been meaning to do some work on a mobile app with my Sister but other work and family commitments kept me hectic.

The Zeitgeist Movement (activism):

  • Helped put together the Aug Newsletter
  • Helped Cliff with the global Zday 2020 event, despite me being in the Philippines and it was in Iceland.
  • Did some work on the TZM global website.
  • Helped remove someone from a position of power whose actions were both bad for himself and the community.

Gaming ~200hrs worth:

  • Destiny 2 – I played about 46.4hrs of this, but most of it was in 2019 with only a tiny bit at the start of 2020.
  • Doom – 25.6hrs played back in January whilst waiting for Xavier to be born.
  • Borderlands 3 – 12hrs it didn’t really draw me in
  • MarZ: Tactical Base Defence – 2.5hrs of a turret defence game that wasn’t as mindlessly fun as I was looking for.
  • Reprisal Universe – 10.7hrs played of the spiritual successor to the first Populous
  • Doom Eternal – 33hrs in July.
  • Terraria – 3.3hrs according to Steam, but it certainly feels like I played it for a week and enjoyed it, but wanted to play Minecraft with RTX enabled instead (but that wasn’t released to the public yet, just teased)
  • Serious Sam 4 – 23.4 hours of fun. I loved this mindless game. Played in October
  • Star Wars Republic Commando – 6.4hrs An old game I was playing a little until Cyberpunk.
  • Cyberpunk 2077 – 75.2hrs played since it’s release, 41.8 of those hours in the last 2 weeks. It’s a great, immersive game similar to GTA but in the future and has Keanu Reeves. I’m aiming for 100hrs with this.

Health:

  • 1,758,072 Steps for the year. 1,499.7Km walked/run for an average of 4,817 average steps a day (less than I’d like the aim is 5,200).
  • Lots of running. Some great trips a decent number of 12km runs and even a 20km run. A total of 524km run, with an average of 10km a week.
  • Got a skin rash, likely triggered by my latex allergy. Also a ringworm like fungal infection.
  • Very little in the way of colds and flu this year. I remember feeling a little under the weather for half a day. The face masks and now face shields are helping.
  • Had some issues with emotional

Articles and videos:

I’ve posted 8 posts on my kublermdk.com blog if you include this, plus some posts to Medium, most I cross posted.

The main ones being:

Travel and experiences:

Obviously it’s hard to get out with Covid19 and a baby boy so we only did 1 main trip.

A trip to Amami beach resort in Puerto Galera for my Birthday. I still need to write up info about travelling in a time of Covid. Thankfully it was a great trip and Jen organised videos from my family, which made me cry.

Deaths:

  • Mario Matiev – He was a friend and fellow TZM member. We’d been Skyping regularly for over a year when he finally died of kidney failure after being on dialysis for a rather long time. I miss Mario and he was working on some great stuff.
  • Paul Parker-Benton – Technically my older brother-in-law but someone I only met a handful of times. Still, his passing affected the family and it was weird seeing a funeral via live stream.

Items and things:

They say that for a happy life you should focus on experiences not materialistic goods. But when you can’t travel much and you live and work at home in a one bedroom unit shared with your partner, baby boy, dog and fish, sometimes things can be useful.

  • New Huawei GT2 Pro smartwatch. This was a great Xmas present from Jen and great timing as my Garmin Forerunner 235 died between Xmas and New Years Eve. The watch doesn’t just track my heart rate but also stress (based on Heart rate variability I think?), it has nicer sleep graphs, even if it’s opinion is that we should be sleeping from 10pm until 6am. The graphical touch display is awesome, the watch has it’s own speaker and I can play music on it or even use it as a mic source for calls (I think, haven’t tried this and I don’t think it works with Whats App, etc..). The watch has 2GB of space, even has a calculator app, torch mode, Air pressure, Compass, etc.. Jen had to sell off her old phone and use all her money from selling bags and things, but it is a very elegant and nice watch. It helps that I got her a Garmin Fenix 5 smartwatch.
  • DJI OM4 smart phone Gimbal – I got this a few months ago for capturing better stock footage but also for filming family stuff with Xavier, etc..
  • eBike – Jen organised a decent enough eBike also for my xmas present
  • Live Broadcast microphone setup – Jen got me a full microphone with pop filter, sound mixing deck with sound board and the ability to make people’s voices sound like chipmunks amongst other things. I hope to do some good audio recordings with this one.
  • Washing Machine – After the full lock down started just after we got Xavier home we couldn’t go to the nearby laundromat and had to wash our clothes by hand. So getting a Washing Machine was one of the first big things we did after the Covid19 hammer was lifted (and replaced with the dance).
  • 10TB HDD – I got a WD RED 10 Terabyte hard drive to keep some of my footage safe. Along with the 8TB drive, some 4TB drives and the files I upload to BackBlaze, Degoo, Dropbox and Amazon S3 to ensure I have copies of my important data.
  • A big 72L fish tank – We were given a big aquarium for free a month or so ago as the people who had it moved out and left it behind. It took us a while to get the bacteria working properly and lost some fish in the process as we were too eager to buy more. But it’s working well now.

Others:

Was affected by 2 Typhoons barely a week apart. One when we were at Amami beach, the other when we were back home but there was major flooding and the power went out both times. We somewhat slept through plenty of other typhoons but those two were more than just a storm to keep inside for.

During the strict lockdown time we had our toilet cistern spring a leak and our bathroom sink kept trying to fall off. It was nearly a month of dealing with it before stores opened up that contained what we needed to fix it.

I still am worried about the sink falling forward, but really want to take the toilet working for granted as having to bucket water into it just isn’t fun.

Hopefully 2021 will be an interesting but less onerous year. As I posted on Facebook, I hope it’ll be less like a bad sequel and more like Endgame was to Infinity War, a great end to a 2 part series. Although with the way US politics is going it might be more like Harry Potter and the Deathly Hallows.


Some videos of the year:

Google AutoML Prediction with a Google Cloud Storage source

As per the Gist https://gist.github.com/kublermdk/0b8c1f6173e5b121e5aee303160fa3f3

<?php

// --------------------------------------------------
//   Example Google Cloud AutoML Prediction
// --------------------------------------------------
// @author Michael Kubler
// @date 2020-10-07th
// This is a cut down gist of what you need to
// make a Google Cloud AutoML (Auto Machine Learning)
// prediction request, based off an already uploaded
// file in Google Cloud Storage (GCS).
//
// The main point is that the payload to be provided
// needs to include a Document
// the Document needs to have an DocumentInputConfig
// The DocumentInputConfig needs a GcsSource
//
// Those things took longer than they should have to
// find and work out how to use.
// The Documentation is auto-generated and hard to
// understand.
// Semi-Useful links:
// https://cloud.google.com/vision/automl/docs/predict
// https://googleapis.github.io/google-cloud-php/#/docs/google-cloud/v0.141.0/automl/v1/predictionserviceclient
// https://cloud.google.com/natural-language/automl/docs/tutorial#tutorial-vision-predict-nodejs

use Google\Cloud\AutoMl\V1\PredictionServiceClient;
use Google\Cloud\AutoMl\V1\AnnotationPayload;
use Google\Cloud\AutoMl\V1\Document;
use Google\Cloud\AutoMl\V1\DocumentInputConfig;
use Google\Cloud\AutoMl\V1\ExamplePayload;
use Google\Cloud\AutoMl\V1\GcsSource;
use yii\helpers\VarDumper;

// -- Things to change
$autoMlProject = '186655544321'; // The ProjectId - Set this to your own
$autoMlLocation = 'us-central1'; // For AutoML this is likely to be the location
$autoMlModelId = 'TEN15667778886635554442'; // The modelId - Set this to your own
$autoMlCredentialsLocation = __DIR__ . '/google-service-account.json'; // Set this to where ever you set your auth credentials file
$gsFilePath = 'gs://<bucket>/filePath.pdf'; // Obviously set this to your file location in Google Cloud Storage

// -- General setup
putenv('GOOGLE_APPLICATION_CREDENTIALS=' . $autoMlCredentialsLocation);
$autoMlPredictionServiceClient = new PredictionServiceClient();
$autoMlPredictionServiceFormattedParent = $autoMlPredictionServiceClient->modelName($autoMlProject, $autoMlLocation, $autoMlModelId);

// -- Setup the request
$pdfGsLocation = (new GcsSource())->setInputUris([$gsFilePath]);
$pdfDocumentConfig = (new DocumentInputConfig())->setGcsSource($pdfGsLocation);
$pdfDocument = (new Document())->setInputConfig($pdfDocumentConfig);
$payload = (new ExamplePayload())->setDocument($pdfDocument);

// -- Make the request (Here we actually do the prediction)
$autoMlFullResponse = $autoMlPredictionServiceClient->predict($autoMlPredictionServiceFormattedParent, $payload);

// --------------------------------------------------
//   Output #1 - All as JSON
// --------------------------------------------------
// You've got a couple of options now, you could return the full set by outputting / returning the serializeToJsonString response
echo $autoMlFullResponse->serializeToJsonString();

// --------------------------------------------------
//   Output #2 - Get just specific fields
// --------------------------------------------------
// Or for this example you might only want the payload[i].displayName and payload[i].textExtraction.textSegment.content
$payload = $autoMlFullResponse->getPayload();
$autoMlProcessedResponse = [];
foreach ($payload->getIterator() as $payloadEntry) {
    /** @var AnnotationPayload $payloadEntry */
    $autoMlProcessedResponse[$payloadEntry->getDisplayName()] = $payloadEntry->getTextExtraction()->getTextSegment()->getContent();
}
echo VarDumper::export($autoMlProcessedResponse); // PHP array format, you'd probably want to JSON encode it instead

// NB: You'll likely want to convert this to a class and provide the $gsFilePath in a method and return the expected response not output it

Reinvigorating TZM

At a meeting last night, Friday the 10th of July 2020 about 15 TZM members had a discussion on Team Speak about trying to reinvigorate the movement.
There was lots of ideas. But a couple of people’s suggestions were on approaches to working out the best option instead of just ways to make TZM great again.Aaron Frost pointed out the need for the Scientific Method and Erykah pointed out how we need to do a post-mortem style review to work out what went well and what didn’t.
Victor tried getting people to sign up to his proposal of doing face to face, street activism and only doing that. Seeing anything else as a distraction that should be shut down.There was a suggestion that we need to go back to the old, more authoritarian organisational structure. A so called “return to the good old times”.But Kees pointed out the Google trend for “Zeitgeist Movement” which is similar to the graph that was in my head, except it drops off much more extremely. It shows that there’s now only 1% or less interest in the movement compared to at the start.
Some suggestions included creating more videos and media and I know one person organising a group working on Podcasts.Personally I think what matters the most is Juuso of Koto Coop who is creating an actual RBE aspiring community which is work towards the actual transition instead of just getting the ideas out.
In terms of core members the movement definitely had a lot of people burn out and do their own thing for the last few years. We also have a habit of burning out at least one good member when they do a global Zday event. Casey, Franky and likely this year it’ll be Cliff.
Late 2018 is when it feels like the movement was at it’s most fragile and very nearly disappeared. But thanks to people like Juuso, Mark, Cliff and the Discord community we managed to keep it going. I personally credit the tenacity of Mark for keeping the global meetings going and making them easy, open and transparent.We also used the opportunity to re-organise the movement. There used to be a pyramid hierarchy of communication. Local chapters would report to the national coordinators who’d report to the Global Chapters Administration. There’s now no longer a central gate keeper group like there used to be.  It’s a lot more distributed. Partly based on my Reorganisation doc, plus some other peoples ideas.
Different people have different areas of responsibility and in most cases there’s different groups who help run different projects.
Myself I’m the main person with access to www.thezeitgeistmovement.com and as such website updates are something I prioritise. I can also post on Facebook so I’m usually involved in things like the Zday events which need updates and promotion.
There is a team responsible for moderating the Discord server, another group who deals with the main Facebook page but again others who deal with a lot of the other Facebook groups. There’s Telegram, Team Speak and more.
It seems that especially during the global Covid19 pandemic there’s been an increase in people interested in the Zeitgeist Movement.That’s not surprising given the fact we are being forced to go into a form of economic hibernation and because the existing capitalist monetary system doesn’t support that there’s some room for change. Something we’ve been wanting for a decade.
So with people interested in making TZM great again there’s a few things to consider:Firstly there’s a question of if reviving TZM is a good idea

As the Zeitgeist Movement is about a systems perspective take on transitioning to a Post-Scarcity society (e.g NL/RBE) using the Scientific Method, Sustainability, Access Abundance, Automation and Technology and the like.
Yes. I think the movement has an important place. It has the potential for far greater long term positive change than Occupy, Extinction Rebellion, Oxfam or even the Red Cross. If you’ve watched Zeitgeist Moving Forward and understand the train of thought then you’ll understand why.
Secondly is how would you go about it.My proposal would be along the lines of “We need to reinvigorate the movement in order to help transition to a post-scarcity society. To do that we need to know why more people aren’t more active in the moment, work out what what we are missing and what is the most effective activities we can do

Just from the meeting alone we have some ideas of what might draw people in, like:

  • Face to Face street activism – Although not during the current Covid19 issues
  • More online media. Videos, podcasts and the like – There’s a great Podcast team that’s being created.
  • Online community spaces – We have FB, Discord, Telegram and the like, so this is mostly taken care of. Although the website needs a bit of a content overhaul.
  • More clear messaging – There was a project by Cliff creating new explanation videos which started this but has taken a backseat whilst he organises Zday.
  • More consistency – The example given was of all chapters having the same making scheme. Personally it sounded a bit OCD and I think we need to have locally appropriate diversification.
  • A change in org structure – Note that this has already been done, just not heavily communicated.
  • A change in target audience to be less conspiracy theory based – This is an interesting one and would require most filtering measures which go against the movements ethos of anyone with a good understanding of the concepts being a member. I suspect spinoff groups could add their own filtering in better ways.
  • A practical, physical manifestation of a transition. E.g Koto co-op.

I’m sure there’s plenty more ideas.

Obviously which tasks people take on depend on their personality, skills and interest. So there’s no one size fits all approach.

It’s likely we need to do a review of historical events until now and if possible interviews with people who are no longer members and people who don’t know about the movement and try working out some experiments to see what is actually effective.

One thing PJ said about the movement is that it’s excitement based.

Being an active TZM member I’ve of course talked to many people about the movement and ideas and something I get very consistently is people saying “So you’ve been around for 10 years and what have you achieved?”

 As TZM is about promoting the ideas of the NL/RBE we explain how we’ve reached lots of people. Millions of views on the main videos. Large amounts of media content, lots of chapters, events and activism.

But people want to know what steps we’ve made towards the transition.

As per my Price of Zero transition talk. That’s where the Crossing the Chasm marketing information comes in useful to understanding why so many people are asking this question. Only a very small percentage of people are the innovators and early adopters of new concepts. Most are practical minded. They will join an RBE aspiring community to get things done they can’t under a capitalist society but they won’t go building the initial prototypes.

So based on my current understandings I think one of the most powerful things we can do is help people who already know the concepts to know we are making steps towards the transition. I’m working on a 5+25 year transition plan of my own.

But in the mean time there’s Koto Coop which is just starting, Kadagya in Peru and a handful of proposals which need resources to get off the ground.

Still, doing the experiments is important and I think we need to work on the metrics which define success.

It is about more members in some specific community (e.g Facebook, or Discord)? That’s an indicator of marketing not actual transition progress.

Is it about the Google Trend line of Interest Over Time going up? This would likely indicate more people interested in knowing about the movement. But partly the virality which helped spawned the initial interest was shaped by the cultural environment which has changed.

Is it about how many political policies are altered, or political parties voted into power? Not likely as that’s not systemic change.

Ideally we’d have the Zeitgeist Survey Project to have a better handle on the actual change in cultural zeitgeist and also some metrics for tracking systemic change. But that’s another project to work on and needs lots of help to get started.


KotoCoop: https://kotocoop.org/about/model/

Google Trends: https://trends.google.com/trends/explore?date=all&q=zeitgeist%20movement

–Michael Kubler
Email: michael@zeitgeist-info.com
FB: @kublermdk

Advanced Filtering with MongoDB Aggregation Pipelines

This is a repost from https://medium.com/@kublermdk/advanced-filtering-with-mongodb-aggregation-pipelines-5ee7a8798746 although go read it there so I can get the Medium $$, because it reads better.


For a project Chris Were and I have been working on we discovered a great way of doing advanced queries using MongoDB’s Aggregation pipelines with the use of the $merge operator in order to filter down to a specific set of customers based on their demographics and purchasing behaviour.

For those that don’t know MongoDB’s aggregation pipeline is much more sophisticated and at least for me, also more intuitive than filtering using Map-Reduce.

The project is part of the APIs and admin control panel backend that powers a mobile app for customers who are shopping.

We know things like the customers age, gender, and state.

We save some of the customer’s transaction data, including the storeId, an array of products and an array of product categories. Although only the last few months worth.

We have an admin control panel where the managers can create a set of filters to select a specific customer group, they can then then send push notifications to those customers, assign them coupons or send them surveys.

Except for the number of days ago, the entries allow for multiple selections. e.g You can select all age ranges, just one, or a couple of them.

Although it’d take you a really long time in the UI to select almost all of the products or even the categories. Hence we have both have and haven’t purchased / shopped at versions.

Example filters:

Age: 0–20, 21–30, 31–40, 41–50, 51–60, 60+

Gender: Male, Female, Other

State: South Australia, Victoria, Queensland, New South Wales, Western Australia, ACT, NT

Have Purchased Products [X,Y, …] in the last Z days

Haven’t Purchased Products [X,Y, …] in the last Z days

Have Purchased from Product Categories [X,Y, …] in the last Z days

Haven’t Purchased from Product Categories [X,Y, …] in the last Z days

Have Shopped at Store(s) [X,Y, …] in the last Z days

Haven’t Shopped at Store(s) [X,Y, …] in the last Z days

They needed the system to be flexible so there’s also include and exclude versions of the filters.
Importantly the Include filters are AND’d together whilst the Exclude filters are OR’d together. So you can get very specific with the includes whilst applying some broad exclude filters.

An example might be selecting people who have purchased toilet paper and alcohol hand sanitiser in the last 7 days but exclude all people aged 60+ and all people who’ve purchased kitty litter. A notification can then be sent about how there’s a new pandemic preparedness set which is now in stock, or how the stores are being regularly disinfected during the Covid19 pandemic.

Another option could be to target people in South Australia who are 30 yrs old or under and have purchased from the Deli – Vegan meats product category in the last 10 days, but exclude those who’ve purchased the new Vegan burger. Then they can be given a 2 for 1 voucher for the new burger.

With many thousands of products, hundreds of categories and a few dozen stores there’s a reasonable amount to search through. We also don’t know how many customers there will be as the system hasn’t been launched yet.

But the system has to be fairly fast as we sometimes need to queue up customer push notifications whilst processing the HTTP requests from the 3rd party sending us the transaction information.

The important parts of the data structure looks like this:

What we needed after after filtering all the customers and transactions is a list of Customer IDs. We can then feed those into a variety of systems, like the one for sending push notifications, or selecting which people get coupons.

A main limitation was that whilst the database servers were quite powerful, the web servers weren’t. My initial thoughts were to process an bunch of aggregation pipelines, get a list of CustomerID’s and do the merging and processing in PHP, but when there’s potentially 100k+ customers and transactions, Chris pushed to go harder on the database. Thankfully MongoDB is powerful and flexible enough to do what we wanted.

In the latest v4.2 version of MongoDB there’s now a $merge aggregation pipeline which can output documents to a collection and has some advanced controls about what to do when matching, unlike the $out operator.

I worked out that we can do two types of queries. A “Haveselect of all those who should stay and a “Have Not” select of all those who should be removed.

For a “HAVE” Select we output the customerId’s into a merged results collection for the customer group with an extra {selected: true} field and bulk delete those without the selected field, then and bulk-update and removed the selected:true field.

For the Have Not’s we select all the customerId’s of those we don’t want, set {excluded:true} and bulk delete those with the field.

Example of Include and Exclude filters

This is an example of the UI for setting the filters. The approximate customer’s is based upon some randomly created data used for load testing… In this instance 17k customers and 340k transactions.

The UI creates a set of filters with the values of things like the productId’s, but the PHP backend, using the Yii2 framework, Mozzler base and some custom code, does some parsing of things like ages. e.g From the string “0–20” to the current unix “time()” to “time() -20 years”. Similar changes are done on the backend to convert something like 60 days ago into a unix timestamp relative to now.

I was going to do an aggregation pipeline for each filter. However if there’s 10 filters that could be a lot of work. MongoDB seems to be better at having somewhat complicated $match queries (with some good indexes) but not so good at running as many aggregations.

Chris then suggested we merge the aggregations and I realised that doing it the following way actually works out perfectly and we only need a max of 4 aggregations. I’m sure if the Exclude filters weren’t considered a logical OR or the Include’s were consider a logical AND of the filters then things would be different.

Aggregation pipelines

1. Customer Select (Have):
Include filters for a Customer’s Age, Gender and/or State

2. Customer Exclude (Have Not):
Exclude filters for a Customer’s Age, Gender and/or State

3. Transaction Select (Have):
Include filters for Have Purchased Products, Categories and/or at Stores
Exclude filters for Haven’t Purchased Products, Categories and/or at Stores

4. Transaction Exclude (Have Not)
Include
filters for Haven’t Purchased Products, Categories and/or at Stores
Exclude filters for Have Purchased Products, Categories and/or at Stores

From the admin control panel UI we’ve grouped the filters into the Include or Exclude and have an array of them.
Because things like 7 days ago needs to be converted into unix timestamp based on the current time, we need to update the aggregations dynamically, hence using MongoDB Views wasn’t really possible.

On the backend I wrote a Customer Group manager system for grouping the filters into the different categories and merging them together.
Whilst the actual queries we did were a bit more complicated than shown below because we aren’t saving the state as a string but a postcode so have a bunch of ranges for them, what we do is very similar to the example aggregation below, based on the filters in the UI screenshot:

The aggregations in the above Gist should have enough comments to explain the actual steps in detail. But it’s expected you’ve used MongoDB’s aggregations to have some idea of what’s going on.

Some points we discovered:

  • The $merge filter lets us put all the data into a merged collection and we can iterate over the results using a cursor, use the collection count and other things to make it both easy and very scalable. The merged collections are effectively free caching.
  • It was very powerful to always be doing a $match (select) query first. It’s fast with the right indexes and powerful with the excluded or selected: true and doing a bulk delete / update.
  • Merging into the Have / Have Not filtersets on the Customer and Transaction models mean there’s a maximum of 4 pipelines that will be run. Although if the Exclude filters were an AND not OR between them, then this might not be the case.
  • An edge case is that we have to add in all customerId’s as a first pass if there wasn’t a Customer Have pipeline so that customers who don’t have any transactions could still be returned, or so that if there’s no filters then it selects everyone.
  • I also developed some tweaks which let us run a set of filters on just a single customer or small selection of customers to know if they are part of a customer group. This is especially used when a new transaction comes in. We point to a different (temporary) customerGroup collection in that case. Obviously querying against a small set of customerIDs makes things much faster.

The end results are good. I get around 600ms when querying against 20k customers and 180k transactions with the basic Vagrant VM on my laptop. Although that is random data I generated just for load testing.

We are still waiting to see what this will be like in production.

Let me know if something doesn’t make sense or if you want more information.

Via Negativa

Via Negativa translates to “by removal.” Taleb argues that a lot of problems can be solved by removing things, and not by adding more. In decision making, if you have to come up with more than one reason to do something, it’s probably because you’re just trying to convince yourself to do it. Decisions that are robust to errors don’t need more than one good reason. You can observe the beneficial effects of Via Negativa effects in a vast number of fields from Medicine and Diet to Wealth.

Copied from: https://anantja.in/antifragile-things-that-gain-from-disorder/

This mental model wasn’t in the main list of mental models I often look at https://fs.blog/mental-models/ but it’s one I’ve come across before and wanted to point out to people so thought I’d post it here on it’s own.

In this case, I’m thinking about using Via Negativa for removing stupid people from a group.

Evernote lost my work

TLDR: Evernote doesn’t seem to backup your work whilst writing. It doesn’t save until you’ve actually exited the note. So it’s highly vulnerable to your phone / tablet dying.

The Story

This afternoon I got my Android Tablet out and started to write up my weekly review. I haven’t actually done my review for the last month, so there was a lot to write in, like how my Baby Boy is developing, issues I’ve had with sudden sciatica in my back and some of the craziness of dealing with the Corona virus reactions and lockdown.

It’s stuff I can re-write, but won’t. I certainly won’t be using Evernote to do so.

Actions

I started by duplicating an existing template I made recently, then I renamed it, sat down for what felt like 45mins and poured words onto the screen.

My tablet is a little old, it’s a Samsung Galaxy Tab S2 and the battery has a habit of dying. Which it did. A notification just appeared to say the battery was at 15% and then suddenly the screen went blank, then it rebooted. I didn’t think too much of it, figuring that my tablet was connected to the Internet and should be both saving locally and to the cloud as I was writing. It was fine if I lost the last minute of work.

But instead I lost probably thousands of words.

If I was writing my Novel I’d be livid. It’s just not acceptable that it’s not automatically backing up. It’s a mobile app, not a desktop app so I shouldn’t need to [Ctrl] + [s] save every few words as I normally do when working. Especially when I’m in the flow and just actively writing I don’t want to think about saving.

Switching to Pure Writer

I’ve been burned, so from now on I think I’ll be doing my initial writing in Pure Writer before copying over to Evernote. I still like Evernotes syncing across devices and the way it works, but I now loathe it doesn’t have an autosave whilst actually writing.

Killing Windows Night Light

O.M.F.G I finally found out why my colour grading has been so off. Windows 10 “kindly” enabled it’s Night Light mode and made everything more Red in the evening.

Night Light mode is the same as F.Lux or Twilight Mode it puts a Red colour over the top of your screen to reduce the amount of blue light, this is meant to help you go to sleep easier.

However, when doing any colour grading work it completely throws off your attempts. You need to disable it in order to do any Photoshopping, Video editing or anything to do with colour grading.

You can go to [Settings] -> [Display]
Then check that [Night light] is switched to off.

Then go to [Night light settings]

In the Night light settings ensure that the Schedule is set to Off and that the Colour temperature at night is all the way to the right and thus is white.

Unfortunately it took me way too long to realise what was going on and why. When I colour corrected a clip and it looked white to me, the Davinci Resolve colour scopes looked like they were out. I’d notice what looked like a bit of Red on the monitor, but would just tilt it until it looked fine. Which seemed fine as I had an X-Rite i1 Display Pro colour calibrator and was using a 4K monitor.

Note that there’s also a few other apps which can cause a colour tint. On my Asus Republic of Gamers laptop the Armoury Crate app includes a Featured app called GameVisual which also likes to do some colour temperature changes of its own.

Hopefully this helps others with some similar issues. Let me know of any other apps which causes problems.

Things I want to teach my children

Here’s a collection of things I’d love to teach my kids:

One of the most important things is how to be happy and successful. The secret to happiness being more than just having low expectations and being happily surprised.
Having a meaningful life where you are working towards a unified purpose is important.

Another thing is not to take on too many things. I’ve found it hard to say no, because I can see that there’s not enough skilled people in the world trying to help and that I can see the potential in many of the projects I come across.
But by making a list of 25+ things you want to do and focusing on the top 5 you can hopefully keep your focus.

Note that to know your purpose and thus how to prioritise what you want to do in life you should read Stephen R. Covey’s book, The 7 Habits of Highly Effective People which will also teach you to do things like balance production vs production capacity as well as taking different views of your life and do decent planning

Yii2 Swiftmailer 0 Auth exception

If you get the error Message: Failed to authenticate on SMTP server with username “****” using 0 possible authenticators

Then try to remove the username and password from the configuration file.

Context

When using the Swiftmailer, a common PHP emailer
This example specifically talks about the Yii2 configuration file, but likely applies to other frameworks.

Here’s an example of the offending config

config/web.php (or console.php or a common.php file if you merge the two).

[
'components' => [
  'mailer' => [
    'class' => 'yii\swiftmailer\Mailer',
    'transport' => [
        'class' => 'Swift_SmtpTransport',
        'plugins' => [
            ['class' => 'Openbuildings\Swiftmailer\CssInlinerPlugin']
        ],
        "username" => "smtp-auth-user",
        "password" => "*****",
        "host" => 'exchange.local',
        "port" => 25,
        ]
     ]
  ]
];

The exception seen was

Message: Failed to authenticate on SMTP server with username “….” using 0 possible authenticators

This exception caused major headache.

After investigation it turned out that removing the username and password from the transport caused it to work.

It seems that the server we were on was in a corporate environment and SMTP authentication was disabled but Swiftmailer was trying to authenticate and failed.

Bonus – Enabling SMTP Logging

[
'components' => [
'mailer' => [
'class' => 'yii\swiftmailer\Mailer',
'enableSwiftMailerLogging' => true,
'transport' => [
'class' => 'Swift_SmtpTransport',
"host" => 'localhost',
"port" => 25,
],
],
'log' => [
'traceLevel' => YII_DEBUG ? 3 : 0,
'targets' => [
[
'class' => 'yii\log\FileTarget',
'levels' => ['error', 'warning'],
],
[
// Log the emails
'class' => 'yii\log\FileTarget',
'categories' => ['yii\swiftmailer\Logger::add'],
'logFile' => '@app/runtime/logs/email.log',

],
],
]
];

With the above config you should now see detailed logs in the runtime/logs/email.log file.