Jen’s Cyst and Appendix issues – Our Medical Experience

TLDR: Jen was sick. Getting diagnosed was harder than it should be.

Summary

Jen was rolling on the floor in pain, had major bloating, some fever and other issues.
We went to the hospital believing it to be appendicitis.
The clock starts ticking. You want to be operated on it within 36hrs to have a decent chance of survival.
Got Blood and urine tests, then Ultrasound. The results were inconclusive. The gas in her stomach stopped them from seeing the appendix.

Talked to surgeon, next step was a CT scan. We couldn’t get scanned until the next morning.
Used the last of our money, even after being loaned some from a friend, on getting a Covid test so they could operate on her.CT scan results finally arrived.
The appendix is prominent, but the main issue looks to be a complex ovarian cyst.
Need another Ultrasound to be sure.
We’d run out of money, but thankfully my clients let me invoice unexpectedly for the work already done and they paid immediately.
The International transfer took less than 24hrs which seems to be a record time.
We saw that the money arrived at 4:15pm but the ultrasound room finished operating at 4pm, so tomorrow we’ll go get that.
My prostate also seems to be acting up. So we might have to go through a bunch of the same stuff again, but for me.

The Full Story

A quick note that at the end of this story I reflect on the Capitalist system we have and how it doesn’t prioritise human health and wellbeing.

So myself, my life partner Jen and our 2yr old boy Xavier are night owls. We don’t normally sleep until 4 or 5am.

We begin this journey at 4am on Tuesday the 18th of January 2022. Our main theme for this year is savings. We used up all our financial savings last year when Jen’s Mum had a stroke and later died from Covid, or at least something similar to the delta variant. Of course to have savings your need to make money so for her part Jen has setup the spare room with her bags and clothes for doing live selling, it’s her first time trying to sell bags via Facebook live. I’m busy in my room and Xavier is watching his cartoons in our room.
Jen starts to feel a bit of pain in her stomach, but is both excited to be selling and a little stressed. She’s prepared for weeks for this, she’s watched nearly 100 hours of other live sellers.
Of course she starts without telling me. It was just meant to be a quick test run, but people start watching.
I notice what’s going on and sneak in and setup the ring light and a bit later a more brighter light.
At 5am she’s still going and I’ve been looking after Xavier for a while and we are all getting tired. I let her know it’s time to wind down. 10mins later I give her a 5 min warning as she’s selling.
She stops, but is on a high. She’s sold a bag had 18 people watching when she stopped and apparently over 200 views in total. Not bad for the middle of the night.
We finally settle in to sleep, Xavier, who’s normally the one wanting to stay up forces us to turn the lights off. It was cute. Just as we are closing our eyes Jen complains about her stomach. I assume it’s the nerves and stress or maybe Xavier kicked her.

I put my eye mask on and ear buds in. I need a good night’s sleep so I can do some good development work. I’m the only back end developer for 3 main projects.

At 7:45am Jen wakes up, clutches her stomach and goes to the toilet.
She comes out a few minutes later and instead of going back to bed and cuddling me she slumps down by the wall around the corner.
I’m still doing my best attempt at being able to get back to sleep fast, so I hazily ask if she’s OK and hear some pained response.

I get up, and see she’s now sprawled on the ground and seems to be crawling away.
I take my eye mask off and pull my ear buds out and she’s already halfway out the door.

I’ve not seen her in such pain since giving birth.

Her stomach is bloated. She’s not in a good state. So I giver her a mefenamic tablet so ease the pain and maybe any swelling.
I order a Grab (the Asian equivalent of Uber) and we get Jen’s nefew Mac Mac to look after Xavier.
We go to the same hospital Xavier was born at and there’s no one else at the Emergency Department.
After a quick message to my Dad who’s a VET, we are worried it’s appendicitis. She’s got nearly all of the symptoms, including the one about how pressing on the stomach and the quickly releasing the pressure hurts. She felt every bump in the car ride there.
They do a urine and blood test.
The Mefenamic seems to have kicked in and the pain has subsided to just when she moves a lot.
We waited an hour, got the results and found out she’s got pus in her urine and there’s higher than expected Leukocyte count (white blood cells).

Blood Test Results
Urinalysis

Just based on the tests they’d expect she’s got a bad UTI. But the specifically localised pain in her lower right abdomen and bloating indicate it’s likely appendicitis.
We get an ultrasound done. It takes a bit of waiting around because they haven’t opened yet and when they do, they need Jen to drink enough water so it feels like she needs to pee. I rush off to the mall nearby to get some water… and the last of the money we have out, so we can cover the cost of the ultrasound.

Eventually they scan Jen and after waiting a bit more we find out that unfortunately, it’s inconclusive. The gas in her stomach has obstructed their view of the appendix and also some other organs like the ovaries.
I mean, we’ve ruled out liver, kidney, gallbladder, pancreas, spleen and bladder issues, so it’s not a complete waste. But it’s not what we need.

From what we’d read and heard there’s a crucial window to appendicitis. 12 to 24 hours is the most likely time range for getting it operated on, then after 36 hours the chances of it bursting increases dramatically and so do the chances of death.

My step-mum had appendicitis and it took them 2.5 days to diagnose it and it had already ruptured and she needed to spend basically a week in hospital after the surgery with an antibiotic IV drip. We don’t want that for Jen. We likely can’t afford it either.

We talked with the ER doctors who referred us to the surgeon who said it was inconclusive. The signs and symptoms of appendicitis are very similar to things like an Ectopic pregnancy or Stomach Flu (gastro).


Symptoms of Appendicitis:

[✔] Pain in your lower right belly or pain near your navel that moves lower. This is usually the first sign.

[✔] Loss of appetite

[✔] Nausea and vomiting soon after belly pain begins

[✔] Swollen belly

[✔] Fever

[✘] Can’t pass gas

Of all the signs and symptoms the only thing that didn’t click was that she could pass gas.
Although given Jen looked like she was 6 months pregnant, the bloating was obviously bad and we’d soon find that she would have issues passing gas for the next few hours, but eventually she’d be fine.

The Surgeon referred us to another hospital to get a CT scan done.
The thing is, we were pretty much out of money.
I did a google search and found that most CT scans cost in the order of ₱5-10k and even that was more than what we could afford right now.
The hospital we went to, whilst being very nice, was also expensive and they quoted ₱21k.
We walked out of there and sat down outside. We were feeling broken. We’d only gotten 1.5hrs sleep, had been alternating between running around and waiting at hospitals for 4 hours now and there just wasn’t any way we could cover the costs of the scan right now, but without it, they wouldn’t operate (unless it got super worse). Jen’s life was on the line.

So I did some more googling and found a different hospital with a CT scanner that was not too far away. I called and they quoted ₱12.7k. OK, well even if we max out every bit of money we can’t pay for that, but let’s go get the scan done first and worry about that later.

We get to the hospital only to find they need to do another set of lab tests, the BUM and Creatinine tests, these ensure her kidney and liver are working fine and she’ll be able to take the contrast enhancing solution.

They get the samples and tell us that it’ll be 2-3 hours to get the results.
It’s about 1:30pm and we are certainly not waiting around to get the results when we’ve got a 2yr old at home and we are desperate for sleep.
I’m mostly worried that when we get the results we’ll need to have the CT scan done, get the results ASAP and get her into surgery so they can remove her appendix before it ruptures. The clock is ticking, but it could all be done within 12 hours from her first symptoms.

We get home and the pain in her stomach is a low level ache, not a constant piercing sharp pain.
Thankfully Xavier is doing fine.
I contact my main client and ask if it’s OK to invoice right now, in the middle of the project. Thankfully Aaron is fine with it and I send my invoice through. We’ll now have enough money to cover any major medical bills. Although it’s going to be a few days before we can get the money.
It wouldn’t have been a problem but only the day before all of this I booked and paid for my flights to Australia and back. I’ve got to leave the Philippines as I’ve been here for nearly 3 years and they’ll kick me out. Because of all the Covid restrictions and Omicron sweeping the globe it doesn’t make sense to do a quick overnight trip to Hong Kong, Vietnam or Malaysia like people in my situation used to do. Instead I might as well go visit my family in Adelaide and spend a few weeks there, that way I can also return with a bunch of stuff I left at my Dad’s place. But I pushed our finances to the limit when I booked the flights. I used up the last remaining money I had access to on my Credit Card. I emptied out my other accounts and only had AUD$120 sitting there. At the time it seemed OK. I knew one of my clients was paying $800 soon and we had enough money and food to last until that came through.
But then suddenly we’ve wiped out what tiny amounts of money we did have and there’s a major bill. We had allocated money that was meant to be for our rent and was already halfway through that and now had nothing.
At least there’s money on the way, but when I get paid it’s in AUD and then have to do an International transfer which usually takes 3+ days to get into Jen’s account so we can actually have it in Philippine Pesos and buy stuff here.

When I first moved to the Philippines I could use my Credit Card. Of course that expired and my bank sent me a new one. My Dad forwarded it on to me and it arrived over 3 months later, despite it not being Covid times. The postal service has gotten even worse, so when that credit card got cancelled because of a stuff up at the bank, I simply got my Dad to scan the card and I activated it online. It’s still at his place and I’ll pick it up next month. But that card is the only way I could directly convert the AUD $ in my bank account into PHP ₱ without needing to do some International transfer and taking days. Ohh how I wish we could use Crypto or that Gcash could accept my Visa credit card.

Whilst waiting for the lab results we get a couple of hours sleep. I wake Jen up a couple of times to check her phone. We should’ve gotten the results by 3pm, maybe 4pm at the latest, but it’s now 5:30pm, so I call them, they find the results. They’ve forgotten to send the results and she gets them emailed a few minutes later. It all looks fine.
We are ready to go there and get the CT scan and then get admitted into the hospital and hope we can pay at the end of it in a few days, when the money arrives.
I call the radiology department and speak with a guy who says that the schedule is from 9am to 12 noon tomorrow. So I take that to mean that even if we go there now, there won’t be anyone to interpret the results until there’s a Dr doing the rounds from 9-12 tomorrow morning. This is pretty standard. Especially at hospitals we are used to. So we book in for a CT scan at 8am the next day.

I’m nervous. I can feel the clock ticking. What happens if Jen’s appendix ruptures? I’ve read that she’ll get pain everywhere and her fever will be sky high. I’m prepared to rush to the ER if that happens and get her operated on straight away because we’ll know what the issue is.
I wonder. Just how crazy is that? You wait until someone is worse and MIGHT DIE, before trying to help them.

I’d asked a friend and old neighbour, Travis, for money and he thankfully sends us ₱15k. Excellent! We might actually be able to pull this off.
I’d just taken our last ₱7k from the ATM, when I try to get ₱10k out it errors about the daily limit being exceeded. We should be able to get ₱20k out per day. I try the ATM again to get out ₱9k and errors again. Ohh well, I’ll have to get the money out tomorrow.
Whilst I’m near the pharmacy I also get a pregnancy test and some milk for Xavier. The PT comes back negative as expected.

I’d been told that Jen couldn’t eat past 4am and at 3:45am she finally had enough of an appetite to have a single biscuit. She’d only eaten some slices of Apple before then.
We wake up, catch yet another Grab and we’ve now gone past the initial 24 hour time window.
I figure everything is laid out. CT Scan -> results -> operating room. She should be able to have her appendix out before noon, seeing as 4pm is the 36hr window mark.

But when we arrive we get told when there that it’s a 6hr fasting window not 4 hours like I’d been told. We’ll have to wait. Damn. Jen only had a single biscuit and basically hadn’t eaten the whole of yesterday.
I have to get the money out anyway, so I head out and find a weird ass ATM that’s right by the entrance to some big walled off industrial area. The ATM nearly eats my Credit Card. After what feels like 10 minutes it finally spits it out with an error. My guess is that the telephone cable to the ATM has been cut or disconnected.
I jog to the mall, which is over 1km away. Except it’s closed. It doesn’t open until 10am. I’m not used to being awake before 12 noon.
There’s no obvious ATMs that I can see from the front of the mall and I’ve run past here a few times before. We’ve even been in once before and don’t remember seeing any. But pretty much all the malls here have a bank of some sort inside.
I see an employee walk in through a side door and I do the same. I know that if I look the part I should be able to get away with it. I’m desperate, Jen needs an operation, we need the CT scan so she can get the operation, we need money to pay for it. Of course, I’ve still got my face shield on and I’m a white guy who’s looking around desperately.
Before I can find a bank inside the mall a security guard spots me, yells out and sends me back. Thankfully he tells me to go around the right side of the mall. It’s the side I’ve not been to before…. and I find out that the ATM there doesn’t allow withdrawals. Arggh.
Ohh wait, there’s actually a bank of ATMs and our bank’s ATM is here. Yay! I now have ₱10k in cash and head back.
We pay and after the radiology people have finished getting ready it starts.
I get to watch as Jen is put onto a table and it’s moved around in the CT scanning machine. There’s a whirring inner circle that reminds me of the Stargate. It’s not quite as massive a portal as that, but it seems pretty epic. It has its own big power supply in the corner and looks somewhat like an MRI machine that got chopped short.
They do an initial scan and I’m amazed at what can be seen.

Then a phlebotomist comes and injects her with a special contrasting agent. Jen seems like a trooper and doesn’t really say anything, but she’s actually in pain. Her veins feel cold and then burning hot.

The operator makes sure Jen is fine and not having some sort of allergic reaction to the chemicals they’ve just pumped into her body. After checking on her a bunch of times over the next 15mins she’s considered fine and they do another scan and WOW, there’s this animation of the different slices through her abdomen down to part of her legs and you can see the bone and muscle and everything. It’s cool to watch.
Then they inject Jen again. Now her veins are tingling as well as alternating hot and cold.
Another scan.

She’s stuck in there with her hands above her head. She can’t move otherwise it’ll cause issues with the scans. She’s also claustrophobic, plus it’s rather cold in the room, so she’s trying not to shiver.
Then another injection. Her veins now feel like they are on fire. She wants to scream in pain, but she thinks about our sons face. his cleft chin and how he loves running around. She keeps it together.

After about 40mins the whole procedure is done.
Excellent, they say they’ll email the results.

Then Jen explains that she was just told they won’t get the results until around 3pm and that’s with her given priority. Normally it takes 24 hours to get the results.

WTF??? We’ve gone through all of this so that they could have a Dr give an interpretation of the results nearly straight away and now we’ve got to wait 4+ hours? What if her appendix bursts in that time? It’ll be right near the 36 hour mark we get the results. What if she dies??

I ask the Hospital whilst we are there how much they expect it to cost.
For an Appendix operation they are suggesting:

  • ₱100k for surgery (AUD$2,800+)
  • 3-5 days recovery
  • The hospital stay day rate is ₱1,800 for their cheapest shared room and, ₱2,800 (AUD$80) a day for their private Deluxe suite and there’s some other options in between.

There’s nothing more can do here. We go home to wait.
Again we are tired and sleep a bit, but we learn from last time. so call up at 3pm asking for the results, but there’s nothing.
I call at 4:18pm, the Dr still isn’t there.

I’m anxious. I call the Surgeon to make sure he’s ready. Suddenly he tells us that a negative Covid19 RT-PCR test is needed so they can do the surgery. Even if he was to do the surgery without one the anesthesiologist would require it.
WHAT? We need a Covid test for an operation? Yep and he says to expect it to cost ₱5k and take 12 hours. Shit shit shit, ARGGGHHHHhhh. We’ve been at 3 different hospitals, we talked to the Surgeon and a whole bunch of other people and they didn’t mention anything about this.
We’ve got under ₱4k and that’s after getting a loan from a friend. But we know a place up the road, Manila HealthTek which does some fast and relatively cheap tests.
We are now in a quandary. The CT scan results haven’t come back yet, but Google Maps says the swab test place closes soon. Jen is obviously in pain and needs some sort of surgery. I’m sure we’ll need the test no matter what as I’m 90% sure she’ll need the surgery. But getting the test means we will be completely broke.
We catch another Grab and this time take Xavier. He loves being able to go in the car and see everything. We pay for the fastest turn around test which is ₱3,400. We are now broke. We’ve got just enough money for some apples and bananas, but certainly can’t buy something like another can of formula milk for Xavier.

I take Xavier for a little walk to see the trucks nearby and Jen is done rather quickly. There wasn’t anyone in the line.
I order a grab. This will be the 8th Grab ride since it started. I’m grateful that I put some credit on it when I got paid last.
Just as the car is arriving Jen gets the CT scan results. We get in the car and she says it’s not her appendix. Ohh?
It seems to be a Complex Corpus Luteum Cyst by her right Ovary. Although the Appendix is also prominent.
That could mean an Ectopic pregnancy or something else.
Umm, I guess that’s good? Maybe she’s not about to have her appendix rupture but well, we aren’t really sure if there’s still an important urgency or not. If it’s life threatening, not a problem or might cause reproduction problems.
Was her writhing in pain yesterday because she the cyst ruptured? Is Jen pregnant?

We contact her OBGYN and the Surgeon. It’s a complex cyst and her appendix is still prominent in size. So the next thing is a TV Ultrasound. She should also get her appendix removed as a precaution.


Except we have no money.
Also, Jen still has a lot of bloating and gas.

I video call my Dad and he suggests that she use a hot water bottle, gentle massage and buscopan, plus some general pain killers. That helps reduce the gas and by the next day it’s nearly gone completely.
At least with the threat of imminent surgery reduced, Jen can have some food and water. You need to have fasted for 6hrs before an operation so you don’t puke, and because of that I’ve been suggesting she doesn’t eat. But now we relax a bit.

The money from my client comes through. Despite him sending it at the start of the day it seems like it didn’t appear until probably some cronjob ran at the end of the working day, I’m guessing someone manually validated it during the day as well. Although I’ve seen some ETF transfers happen in under 1hr, I’m glad to have at least gotten this. Even if it’s going to take a while to get it converted into Pesos and into Jen’s PH bank account.
That night around 1:30am the negative Covid test results come through. We half laugh and half cry. Thankful we aren’t at the hospital and being rushed to surgery.
Jen is feeling better and does some live selling again. This time she sells ₱10k (AUD$285) worth of bags over the course of 2 hours. Not bad. That’s not profit though, just turnover.

She’s still sick, still has pain in her belly and even bleeding and white strips on her arms from all the needle sites where they’ve taken blood or injected her.
But we go to sleep around 4am.
I wake up 15mins later and need to pee.
I can’t sleep. Then not even 10mins later that I need to pee again.
It’s not a normal peeing sensation. It’s like there’s pee in the urethra tube in a way that simply using pelvic floor muscles doesn’t seem to stop. It also doesn’t feel like it’s from my bladder.
I need to pee again.
Ohh no, is it my prostate? Is it just enlarged or is it cancerous? Or maybe a bad UTI?
How will we afford even a urine let alone blood test? I should get my PSA checked as well as check for pus and other signs of infection.
Will I need anti-biotics? I don’t know. But I’ve got a pain in my groin that won’t go away and Jen has an ultrasound scan to be done and it’s still going to be days before the International transfer arrives.

Actually the initial International transfer only took about 24 hours. Although still only arrived just after the Ultrasound was no longer manned for the day. We’ll go tomorrow and I’ll also try getting a checkup as well.

Ideal Scenario

Ideally this whole situation would be very different in a Post-Scarcity Society, even just based on existing technology we would have had a much better time.
The medical industry here in the Philippines works similar to Australia and many other countries in that it’s partly government funded and partly user pays.
There might be differences, like the PhilHealth system (Philippine Health Insurance Corporation) is equivalent to Medicare in Australia, but focuses mostly on covering some of the costs once you’ve been admitted to hospital. But we’ve been trying our best not to have Jen admitted. Partly because she’s been needed to go to different hospitals for the CT scan and other lab work and partly because Jen is still a little traumatised from the C-section she needed for having Xavier.
The system works well for a monetary based system. The combination of private, public and government funding helps keep things cheaper than other countries which don’t do this well *cough* USA *cough*
I saw a great example recently about how Singapore’s healthcare system works and Australia is pretty high on the list. The Philippines setup seems fine as well

I keep thinking about how things would work in an ideal world. In a Post-Scarcity Society (e.g NL/RBE). I am a volunteer with TZM after all.
When we needed a CT Scan, the costs wouldn’t have been an issue. This is someone’s life at stake, not some plastic surgery or something just nice to have. So priority would be given.
I envisage a future where all your data is owned by you, so the various results and information would have been automatically sent into a Verida like datavault that Jen could easily share with the hospital and myself and in a secure manner. The Drs could easily add notes and information.
In the mobile app it could’ve analysed the symptoms, highlighted the likely outcomes and been helped along by the Drs recommendations. It would’ve indicated a high chance of surgery being need and right at the start would’ve suggested also getting a Covid19 swab test done to start that process in case of urgent surgery. It might even have been able to guess that the Ultrasound would’ve had issues and suggested getting the extra labwork tests needed for doing a CT Scan.
That way Jen would’ve been able to get the lab tests done at the first hospital all of it could’ve been quickly processed. We’d have then been able to search for the nearest available CT scanners rather easily and have a list of multiple hospitals to choose from along with their current availability and scheduling. Selecting one would automatically trigger a car to arrive and pick us up and take us there.

The CT scan could be done and the data could be analysed in under an hour, likely using a combination of AI and a team of experts who can analyse the data whilst working remotely from anywhere.

Even if we weren’t in a Post-Scarcity Society, just the pain of bureaucracy has lead to a snowballing escalation of issues. From the requirement of me to leave the country (a requirement in pretty much every country I know of) to the problems with trying to transfer money and how slow it all takes, to the issues with trying to get and share data. Certainly the financial issues and medical data side of things could be pretty easily resolved with DeFi and some of the ongoing Crypto projects that are in development but I’m sure will take a while to fully go mainstream.

My issue isn’t with any specific country, but with the way human health and wellbeing is secondary. Money and profit are the most important things in our society.
I love to program and film and create content and be in the flow. I’ll do that even when not paid, although maybe I’d be working on other projects.
But I’m very grateful to have a job that’s got flexible working hours and a great pay. I feel very sorry for those who don’t. Who work really really hard and money just seems to slip through their fingers.

2021 Year in Review

Something I wrote about a while ago was doing weekly reviews.
Unfortunately Jen and I probably only did 6 or 7 of these this year with my Oct blog post being the closest to actually writing a review down.

A few things that stick out for this year:

  • We had Xavier’s first Birthday
  • Jen’s Ex said he wanted her back. He’s a violent person, possibly a psychopath
  • We got a big fish tank
  • We moved house to a place nearly 8x larger
  • My hourly wage increased to AUD$80/hr as I moved from being a sub-contractor to the main person in-charge of 3 decent sized projects. Although spoiler: Being sick makes it harder to get the work in and hours up to what would be ideal
  • Now I’ve got my own room to work in, I’ve been better at understanding what it takes for me to do deep work (I’ve been working on a blog post about this but it’s low priority)
  • Jen, Xavier and I have been sick or dealing with emotionally draining issues for the last 6+ months
  • Jen’s Mum dying was a big shock and big deal and disrupted our lives, but also meant Xavier and I have been able to meet a lot of her family
  • We had the seemingly yearly tradition of travelling around my birthday and experiencing a Typhoon a day or two after arriving.
  • It’s been good having Mac Mac here instead of our old house keeper
  • I was good having Popsie here, wee miss him, but hope he’ll get to see his Mum

Just recently

Today I’ve been busy with going through the websites I run and getting a Duplicator backup of the WordPress sites and making sure all the plugins and sites are updated, especially those with the All-in-one SEO plugin which has a major, easy to exploit vulnerability. I also setup Borg updates which has been good.

My Review:

1. Finances

Sent my invoices? Paid my bills? Next bills coming up? Financial snapshot? Assets and Liabilities listed? Working on creating a passive / portfolio based income stream?

I moved from getting paid monthly to directly working with clients. One pays me monthly for the hours I work, but it’s not enough to survive. The main client pays me when the project is completed. Back in October I was still working on a big project which ended up taking about 4 months to complete and I didn’t get paid until the 2nd Dec because it was both bigger than expected but I was also sick or busy with Jen’s Mum and everything else.

It was nice to get some money from Blackbox every month because we got a stock footage sale. It will be good to increase how much we upload.

Income in the last 12 months: AUD$43.8k
Debt: AUD -$2k
Cash / Bank account money: AUD$500
We also blew through all of Jen’s savings which was worth around AUD$5k, so that needs to be paid back. Plus there’s the costs of travelling to Australia and back coming up. The bank account won’t be happy for a while.

Money was a big issue for us this year.
I expect it’ll be an issue next year when trying to travel to Australia if Feb. Then we basically just want to get a whole lot of savings before I try something like working on my startup.

2. Health and Fitness

The last 6 months can be summed up with the word SICK.

  • Around late August I got a stomach bug because our house keeper didn’t wash her hands properly. I needed antibiotics.
  • Just as I was getting over that Jen then got a cold and a few days later Xavier got it. Then I caught the cold. So we looked after each other
  • Our dog got ticks
  • I got some bad hay fever like symptoms which I later found out was from an infection on my skin, possibly related to dealing with our dog so much. On top of that Zyrtec (Cetirizine) anti-histamines would make me really drowsy for a full day. I’m glad that strong hydrogen peroxide on my hand has removed the infection and my hay fever is pretty much gone.
  • I got my first AstraZeneca Covid vaccination dose. Then promptly went for a really long run and felt bad for a day.
  • There was a couple of weeks when everything seemed fine and I was full time productive.
  • Jen’s Mum had a stroke. We had to go up to visit her and spent a week nearby. It was a very sleep deprived week and we all caught a really bad flu from the hospital. Possibly Covid (didn’t have the money to get tested). Took about 2 weeks to recover
  • There was 1 day when everything seemed fine. I felt like I was over the flu and back at full strength.
  • On Halloween night Jen’s Mum died from the flu whilst also trying to recover from the stroke. It was devastating.
  • Jen’s local family visited for 3 days as a mourning ceremony. I ended up sharing a shot glass and getting a stomach bug. Was sick for nearly 3 days.
  • Jen was devastated and an emotional wreck for a long time
  • Our house was found to have termites and the landlord started pulling the ceiling down and spraying the wood with very strong poison. The fumes were very bad and we couldn’t move out of the house at the time as we had no money. They’ve yet to complete all the spraying and ceiling replacement. The rest has been put off until we travel in Feb. Covid19 willing.
  • After 40 days of mourning Jen and her step-Dad went to Pangasinan to bury her Mum’s ashes next to Jen’s Dad’s. She brought back a cold which Xavier and I also caught.
  • There was a day or two when everything seemed fine.
  • I went for a long 14km run and ended up very exhausted for a couple of days.
  • Jen’s family from Pangasinan came down to visit for a few days and we’ve caught yet another cold from them. Still getting over it. Although this time it doesn’t seem too bad… Except
  • I ended up with hemorrhoids. So whilst having all the muscle weakness from a cold I also can’t sit down for very long. But I’ve still had to work between Xmas and New Years.

Thankfully it’s not been COMPLETELY just a time of being sick. I’ve still been working and exercising. At least, exercising when I can. I’ve been able to do a few runs here and there. It’s just every time Jen and I start running we get sick and have to start over again.

3. Family and Friends

Obviously the death of Jen’s Mum was devastating. But I’ve been making sure I try and video call my Mum and Dad and my friend Travis and I have been catching up for occasional games of Minecraft or visiting each other’s place and having all the kids play.

We’ve also had Jen’s family visit a lot this year.

4. Production vs Production Capacity – Balanced?

The focus this year has been on production. On doing lots of paid programming work. We’ve not had a chance to do much stock footage work nor work on ourselves or my own startup projects.

But, whilst I got close to it, I didn’t let myself get burned out trying to work whilst being sick.
I’ve learned to take time to relax. If I’ve been flat out for 3 days straight working or dealing with things and barely had time to myself I’d take a day to just play computer games or spend time doing what I wanted to do and then would be right back at it after the down time. But in years past I’ve not given myself the chance to properly relax into that down time and ended up doing sub-par work for weeks and causing emotional issues.

5. Household tasks and general organisation

Jen’s mostly got this covered. She keeps the house clean. Although I’ll occasionally come along and do deep cleaning. I’m especially picky about glass, mirrors and the like being clean.

I did sort through the cabinet in my studio room this week which was good. It was becoming hard to find anything.

6. Short Term Wins

Learning Vue.js, TailwindCSS and Hugo have been great from a programming perspective. I’ve also been doing some great work with the Yii2 PHP projects and I had been doing some good work with the NodeJS based platform I was developing for the stock footage analysis system. Although that’s on the backburner until I can afford to bootstrap it.

Meeting Jen’s family was another unexpected side effect of everything that happened this year.

Getting some of the interesting projects as part of my work done was great.

Working on the Post Scarcity Podcast was also good, although we’ve not yet completed that, need another episode before we release it.

Setting my room up to be a Podcasting studio with sound insulation around the walls has made my room feel more professional. I want to spend time in here a lot more now.

7. Vision of the future

The shorter term vision of the next few months involves:
Jan Save money. Thankfully we aren’t also being kicked out of our home whilst they deal with Termites as was on the cards.
Feb – I’ll be kicked out of the Philippines on the 24th of Feb as I’ll have been here for 3 years by then. As I’ve got to leave, the aim is to travel with Jen and Xavier to Australia. But there’s financial as well as paperwork and Covid19 restriction issues. I’m especially worried given Australia has been so strict with lockdowns and they’ve recently been hit hard and have over 1.3k cases in South Australia just yesterday. It’s a massive spike for SA and I’m guessing it’s mostly the Omicron variant, but haven’t researched much.
With some luck we’ll be able to go and it’ll be a case of a 2 week quarantine stay at my Dad’s place. With less luck it’ll be a 2 week stay at a Quarantine hotel. With even less luck it’ll be just me going.


I mainly want Xavier to meet his grandparents, otherwise we’d just travel to Hong Kong and go to Disneyland for a few days, which is our backup option, with just me travelling to Vietnam or somewhere like Malaysia for a day or two as the emergency option if needed.
March – If we are in Australia the aim is to spend a month there. 3 weeks with my Dad, a week with my Mum and a week travelling. I’ll still need to be working, but I can work from anywhere as long as I’ve got wifi.
April onwards – Pretty much just saving money. The aim is to have AUD$25k in savings. Which means we’ve got to earn more money and put half to savings. Our rent is 1/4 our currently monthly income. I’m already working at the max of 4hrs/day in the flow (deep work) for 6 days a week when I’m not sick. So hopefully we won’t be getting sick.
We might however also try for another child. We’d love a girl if possible.

Long term Vision


My main long term aim is to help humanity transition to a Post-Scarcity Society.
Jen’s main aim is to raise great kids and have a great family. She’s an amazing Mother and partner.

8. Important but not urgent tasks

Something I’ve managed to do over the last 2 days is rework my servers backup systems.

I have a server with 1TB of HDD space which I used for offsite backups. I had an extensive Rsnapshot backup system setup, however the HDD got corrupted, the hosting provider said it was something about an unstable host, so they moved my VM to a new machine. That was months ago and I’ve been without a good rolling backup system for a while. So I’ve finally set one up and have switch over from Rsnapshot to Borg which includes compression, encryption and deduplication on a block level (so appending a single line to a multi-GB log file doesn’t make a whole new copy).

I’ve then got to get back to work on the TZM.community remake with Cliff.

I’ve also got to finish off my Post Scarcity Podcast episode on Cradle to Cradle.

But all of these things take up my deep work time and means I’m not working

9. Learning and Education (what to direct my learning towards next)

Some books I loved listening to this year:

Podcasts I recommend:

You Are Not So Smart

You Are Not So Smart Podcast website

I also enjoyed the Gradient Dissent podcast about Machine Learning, but that’s a lot more niche.

Compressed.fm is OK, but not amazing. Mostly because I’ve got 10+ years experience in doing web development. However I did learn about some new tools, like Tailwind CSS and want to actually use it from listening.

10. Teaching, Training, Writing and sharing my knowledge

I’ve not done nearly enough of this. But I’ve got a few posts and even videos I do want to make.
Unfortunately I’ve not had time to work on my own projects in the last few months so I’ve not had a chance to do things like the Kubler Code’s live streaming whilst programming on my own projects.

11. Willpower challenges and strength. Or behavioural changes

Mostly just trying to be less distracted by Social media, get more meditation time in and return to good sleeping patterns when possible.

All in an aim to work more so the family can afford to live well.

12. Food/Nutrition

Doing fine with this. Since Jen stopped breastfeeding back in November she’s been serving less food and I’ve also gotten better and leaving food on my plate or others plates and trying to eat later instead of stuffing myself full. So I’m no longer increasing in weight little by little.

13. Gratitude. What are we grateful for?

My family.

Our health (when we aren’t sick).

The fact that I have a flexible, well paying job which lets me work whatever weird hours usually work for me (except when there’s meetings).

Xavier

14. Sleeping patterns and other behaviours?

We are back to our usual night owl routine of going to sleep around 4am and waking up around 1pm. Jen, Xavier and I are used to it. Although there was a period of about 2 weeks where there was some emergency I had to attend to in the morning or something was happening with Jen’s family. But we are generally stable.

15. Jen, Xavier, Maui

I love them all and miss them when they aren’t around.

Well, except Maui our dog ;-p

16. Love life

Yeah, I’m not going to go into any detail about this except to say that Jen and I love each other a lot.


Overall, it was a draining, exhausting year.

Hopefully 2022 will be more financially rewarding and in the 2nd half of the next year I’ll be able to spend more time working on my own projects.

Life Review Oct 2021

Something I do with Jen nearly weekly is a weekly review.

But what I haven’t done recently is document it, let alone publicly.

I’m not going to do a full review, this will be a bit more a stream of consciousness. At least at the start.

Yesterday was my Birthday and I’m now 37 years old of 13,515 days.

My baby boy Xavier is now 652 days old (1.8yrs) and Jen and I have been together for 1,234 days (3.4 years) with me living in the Philippines for 2.7 years of that.

Jen and I regularly say how we will be together for 50 years. It’s in our unofficial contract. We aren’t married and don’t intend to (although might if we need to for legal reasons more than anything else).

In July I transitioned to working with some clients directly on my own instead of being a contractor. This came with a pay bump from AUD$50/hr to AUD$80/hr which is nice, but I’ve got more responsibility now as well.

I did a project from July to August for 100 hours and that was great. I then started the next project which was also meant to be for 100 hours and well, life got in the way and I’ve still not completed it.

Firstly after having gotten paid I then spent a week working on my own project, Gather Together. I’d started it nearly a year before and after lots of issues with Dropbox and other integration issues I switched to a locally run command line system and it worked really well. I can now get keywords for video files. It needs a lot of work and refinement before I can provide it as an auto-keywording service, but it’s already helped Jen do the curation for a whole lot of Stock footage clips.

I’ve since had to jump onto a different side project that my Sister is especially excited about. More on that once I’ve got something to show.

After the week working on my own project there was then nearly a month of getting sick with our house keeper (Yaya) giving me a stomach bug because she didn’t wash her hands properly before making the family breakfast. I eventually needed antibiotics. Just as I was getting over that Jen then got a cold and a few days later Xavier got it, I had to look after them as our Yaya had left for another island in the Philippines due to a death in the family. Then I caught the cold.

Our dog had also gotten ticks during this time and being around his fur so much caused me to get some bad hay fever as well. His ticks were an absolute pain. They were by the carpets in the house, up the walls, in his bed. We spent an hour or two nearly every day removing ticks from him using tweezers until finally using some Detick which really helped… For the next 2 weeks.

I got over the cold and discovered I had some Zyrtec (Cetirizine) anti-histamines. The tablets are meant to last for 24 hours and make you a little drowsy. But they made me REALLY drowsy for the whole 24 hours. I’m glad I took it and was affected over the weekend. But a week later I needed to take it again.

I also got my first AstraZeneca Covid vaccination dose. Jen was still sick at the time and couldn’t take it. By the time she got better they ran out of doses, then we had to go travelling.

During this time Jen’s nephew Mac Mac came down to help out around the house and look after Xavier, helping replace our Yaya.

Then things started to get better. I got back into running, I was working full time hours again and smashing out some good work. I’d finished a migration from AWS’s Elastic Beanstalk AMI v1 to Linux 2 and a whole bunch of other changes that brought with it.

Jen’s Mum’s Stroke

Then around 5pm on Friday the 8th of Oct I walked into the bedroom to find Jen and Mac Mac crying, looking at a video call on her phone of her Mum who’d had a stroke that day and her Sister was about to take her to the hospital.

Within 10mins of knowing what had happened I said “We are going to go visit” and I started packing. I also said that instead of doing a trip for my Birthday, like we usually do, this was my birthday trip. Jen organised the hotel and transport. It took pretty much 24 hours until we got in a car and another 10 hours of travelling to get up to the hotel. But I’d organised for the friendly neighbour to help look after our Dog Maui and another friend to feed the fishes occasionally. Thankfully Xavier slept most of the trip going up there, although Jen was a bit car sick at one point. Jen had maybe 2hours sleep and I had maybe 4 hours sleep on Friday night and then only a few hours sleep in the car on the way up. But we got there.

Jen and Mac Mac went to visit the hospital early Sunday. Then a typhoon hit. OK, not an actual typhoon, but Tropical Storm Maring which included gale force winds and flooding which lasted the whole week we were there.

Some Drone footage of the flooding

The night of the storm I remember staying up a bit, but eventually going to sleep. I woke up when the sun had started to rise and saw the flooding and rain and thought “Ohh, I’ve got to get out there, but I’m really tired” Jen then motioned me to get back to bed and I did. Only to wake up hours later and it seemed like I’d missed the chance and there was almost no flooding. But a few hours later the flooding was back and it seemed like it was tidal flooding.

Some more up close footage of the flooding

Jen’s Mum had an ischemic stroke and hadn’t been treated for many hours. She couldn’t chew, swallow or talk. She can now do those things but 2 weeks later still can’t use the left side of her body.

On the Monday Jen’s Mum got a CT scan and on Tuesday we picked up the results, it was done at a different hospital and on Wednesday morning Jen left the house feeling sick, not having slept well and with only a tiny bag thinking she’d just go for a basic visit. But the Drs discharged her. The hospital had a high number of Covid19 cases and it was more dangerous for her Mum to stay there. As it was on Wednesday Jen and myself started coming down with a cold, likely something she brought back from the hospital the first night she visited. The face masks and face shields only do so much.

I was worried for Jen a lot on Wednesday. So many days with so little sleep and now a cold and she wasn’t messaging me much. Turns out she was dealing with having to pay for the hospital stay, organising Philhealth (Philippines Health, their equivalent to Medicare in Australia), and she went to her Sister’s place where her Mum was now staying.

That cold lasted weeks. I’ve still got a bit of a cough from it and after returning from our trip was still sick for half a week before being able to really get back into work. But I have managed to pretty much get over it.

Wednesday night we slept fairly well and Thursday a lot of the family, her Sisters and their kids, came in a big van Jen had organised and we all went to the beach for the day. It was good, positive fun and Xavier really loved it.

Friday we went home. With Jen, Xavier, Mac Mac and myself. But also her Mum and Step Dad. So now there’s 6 mouths to feed at home (7 if you include our dog).

Talking about our Dog, we’d noticed the day before we left that he had ticks again and came back to find a bunch of decent sized ones on him. The next day I went out and got some Frontline and bam. He’s been fine since. So I’m glad of that.

We were pretty much out of normal funds before the trip and that trip drained our emergency fund to the point we haven’t even been able to cover rent. Thankfully with a nice gesture from our landlord they agreed to take the month’s rent out of our initial bond. Obviously a 1 off thing, but it’s helped us a lot.

We still can’t afford all of the medication and testing for Jen’s Mum. I’ve got a bunch more work to do before my big pay day and well, I spent most of this week stuck with a bug.

We are receiving emails, converting them to PDF and then doing some machine learning processing on them. But instead of selecting the first page it was now selecting the last page. I spent 3 days on the bug. I was going crazy trying to test the different steps, only to find that the newer version of Image Magick was now using the -flatten command to get just the last page of the PDF file and I was already uploading a PDF document. Anyway, I can finally move on and hopefully with a day or two’s work I can finish this current section and then have about a week’s more work.

Programming Work

Here’s a chart of my productivity since August. It’s not nearly as steady as I’d like. I’m looking forward to life getting back to normal.

BlackBox (Stock Footage)

In total, over the last 3 years Jen and I have submitted 870 clips to BlackBox the stock footage platform. Mostly Drone clips, plus some timelapses with my DSLRs, some GoPro underwater footage and more recently I’m starting to upload clips from my phone using a Gimbal.

As I was writing this post I got an email that we’d just had another sale. Yay!

Obviously this is a very tiny amount of passive income. But we are looking forward to expanding the number of submitted clips a lot more now I’ve got the auto-keywording system to help with an initial set and Jen can start doing some work again when not looking after Xavier or her Mum.

The Future

I’ve got a number of things I’m looking forward to.

I’m obviously looking forward to getting paid. Although pretty much all the money will be going to paying off our debts and re-establishing some savings.
I’m looking forward to completing the script for my Post-Scarcity Podcast on Cradle to Cradle.
I’m looking forward to working on a Minecraft hardcore video project.

I’m interested in doing a Kubler Codes Twitch stream maybe an hour a week when I’m working on my own programming projects. The idea came to me this week as I’ve been listening to the podcast Compressed FM.

On the topic of Podcasts. I HIGHLY recommend the You are Not So Smart podcast, episode 217.

YANSS 217 – How brains turn noise into signal, chaos into order, electrical spikes into meaning, and how new technology can expand subjective reality in ways never before possible

This is a seriously good podcast episode. It explains why we dream!
The David’s discuss how the brain is always rewiring itself based on inputs and how after only 60-90 minutes of being blindfolded your visual cortex can start being rewired for use in processing touch, sound and other senses.
So we dream in order to stop our brain from rewiring itself!
That’s crazy, but cool. It’s also interesting how you can learn new senses relatively easily.
David Eagleman has developed a vibrating wrist watch like device where you can program it to provide a new sense.
Maybe have it vibrate according to stock market trends or the weather forecast or if there’s aircraft nearby, or the state of a factory or your Minecraft player that you’ve left AFK.
Maybe you want to wire it up to social media hashtags and get a sensation of society.
Your body will soon learn to feel this new sense and you’ll have ESP!!
I thought it’d be a long while before such tech exists but our brains are so good that most developers like myself could be writing code that’ll give you a new sensory perception today!
That’s epic and awesome. But they keep going and discuss even more. Like how our brains are running in a simulation but there’s different parts of the brain running with different ideas and they can clash. I knew that part from a book I read some years ago, back when I was still living in Australia. But still seriously, nearly everyone should listen to this awesome episode!

One Oasis, Pasig – Building Letters

For those who need to do deliveries or visit a friend at One Oasis, Pasig, Metro Manila, Philippines then here’s the buildings with their labels.

One Oasis, Pasig - Building Letters labeled
One Oasis, Pasig – Building Letters

As you can see from the image, the lettering at One Oasis is a little all over the place.
If you are walking from the main gate (top right of the image, just outside of frame) around the loop to the bottom right, bottom left then out the rear gate you would see:

  • Building A to your left
  • Building B to your right
  • Building C on your left
  • Building D on your left
  • Building G on your right
  • Building E on your left
  • Building H on your left
  • Building K on your right
  • Building J on your left
  • Building I on your left
  • Building F on your right

If you need to get to the One Oasis Hub (green coloured roofs) then you need to exit the compound as those two buildings are separate and mostly focused on the main road.

If you are interested in moving to One Oasis then here’s my review of it:

One Oasis has a bit of a weird building letter layout, but that’s really the only main issue.

If you like apartment living then it’s the best place around.

Easy access to Lucky Gold Plaza for doing your shopping at Robinsons or by walking down to Life Homes.

The pool is currently unavailable due to Covid19 restrictions but is great when you can swim.

The centre circle area and playground are great, children friendly places. The basketball court is often used for exercise classes or ball games (subject to covid restrictions) and the clubhouse is a great place for having big events like birthday parties if you don’t mind paying a little (again, subject to Covid restrictions).

The main loop is almost exactly 500m, so two laps of the place and you’ve already run 1km. Even if you can’t run outside due to lockdown or curfew you can run around here.

You can dry your clothes in fenced areas on the roof, although some ground floor units get some extra space outside (but usually don’t get good mobile data connectivity).

The view of the sunsets from Building I are amazing.

We’ve been living here for 2 years and lived at Sorrento and Ciudad before. Whilst here we’ve raised a family so are mostly moving out because the we need more rooms and want a house that’s pet friendly.

For location, close to shopping, hospitals and quick transport into Ortigas this is an amazing place.

Although a word of caution: Holy Life hospital can be fairly expensive. Mission Hospital is a bit better, although we usually ended up going to Mega Clinic at Megamall a short UV Express ride away.

Units likely cost ₱15-20k / month but actual costs obviously vary by owner.

There’s security guards at the two gate entrances and each building plus some other spots, there’s also security cameras, so you feel safe. If you need repairs there’s a handyman/technician or two around.

Internet is great. Been using Globe 100mbit (VSDL) and there’s Converge ICT fibre available.

The tap water is usually ok to drink but not always so Water supplies are available usually ₱20 for a 20L container, plus one time deposit via Aquabest (you’ll see the special Tricycle riding around just talk to them or phone the service).

Overall a great place to live if you enjoy apartment living. Friendly people, great location, and a great place.

Enjoy!

Open letter to Mackenzie Scott: 5 Why’s with Effective Altruism – Invest in Post Scarcity Solutions

This is an open letter to MacKenzie Scott and anyone else donating money to charities.

Dear MacKenzie Scott,

I am impressed with you signing the Giving Pledge and the sheer amount of money you’ve been giving. The world collectively thanks you.

However, when looking at how you’ve been selecting and aiming your donations I feel there’s some aspects missing to the approach and I think a 5 Why’s analysis would work well.

The 5 Why’s is an approach to working out the root cause of a defect or problem by repeating the question “Why?”. Each answer forms the basis of the next question.
You repeat the process until you come to the root cause (which can often be 5 times). You then apply corrections and fixes at each of the levels.

It’s that last point which I’ve found many charities, groups and organisations are unable to do as they exist only work on a certain level.

An example 5 Why’s:

 The vehicle will not start.

  1. Why? – The battery is empty.
  2. Why? – The alternator is not functioning as the alternator belt has broken.
  3. Why? – The alternator belt was well beyond its useful life, but not replaced.
  4. Why? – The vehicle has not been maintained.
  5. Why? – The vehicle is so old that maintenance costs are too high.

There’s a few different funding strategies you can apply. Broadly speaking I describe them as the pyramid, upside down pyramid, cylinder and spotlight.

The Pyramid Funding scheme

The pyramid focuses most of the money and resources on the first Why and applies less money as you go up the pyramid. In the vehicle example you’d spend the most money on buying a new battery, some on an alternator belt and a little bit towards a service, but barely even browse your phone whilst on the toilet looking at new cars.

Inverted Pyramid funding scheme

The Inverted Pyramid is the opposite of the Pyramid funding scheme. In this case you might first get your car battery recharged and put on an alternator belt, but you’d spend most of your time and money on buying a new electric car. That way you don’t have to worry about all the other issues which will soon get worse and stop it from working in other ways.

The Cylinder / Tube funding scheme is straight forward. You just spend about the same amount of time and money on all the sections. So you might have the technician get the alternator belt changed straight away and the car jump started. You might also buy a car diagnostic system so you can investigate what other issues might arise and fend them off before they cause the car to break down on the freeway, blocking up traffic. You’d then buy an eBike as a backup for those days when your car is in the shop for repairs, which will be often enough you’ll also need a great raincoat for when riding the bike in the rain, plus get a lock for it and it’ll need it’s own servicing occasionally.

The Spotlight funding scheme usually focuses on a single metric, e.g lives saved per dollar spent, which initially seems very admirable. An example was chemically treated mosquito nets in Africa, building water treatment facilities or sending condoms to reduce AIDS. Unfortunately this means there’s often a myopic focus on a symptom but without enough investigation into the other why factors. e.g You hear that the battery is dead and you get the battery replaced. You’ll soon find the new battery dead again and instead of recharging it or repairing the alternator belt you complain about being sold a faulty battery.

As per your post about 384 ways to help:

Some are filling basic needs: food banks, emergency relief funds, and support services for those most vulnerable. Others are addressing long-term systemic inequities that have been deepened by the crisis: debt relief, employment training, credit and financial services for under-resourced communities, education for historically marginalized and underserved people, civil rights advocacy groups, and legal defense funds that take on institutional discrimination.

384 ways to help

I can see you and the team are addressing some of the systemic issues.

However none attempt a system level redesign.

Thankfully with all the due diligence and work you’ve done in investigating the charities it seems that you aren’t using the spotlight funding scheme. You use the word “systems” 5 times between the two posts and the charities selected seem to have a mix of what I would consider the first 2 or 3 why’s with a Pyramid funding scheme. Most of the focus is on the short term problems with some addressing in-system change. But I couldn’t see any that are addressing the root cause of most of these issues.

Obviously not all problems have a single root cause. However many of the causes you were targeting, including Racial, LGBTQ+ and Gender Equity as well as Functional Democracy, Public Health, Climate Change and more are a result of failures of our current socio-economic system. In the car analogy, the focus has been on repairing the car instead of getting a new one and working out how to use it.

Your current focus is especially on those affected by Covid19 so a 5 whys analysis would look something like this:

Q1. Why are so many more people starving, homeless or simply struggling to meet basic needs?

A1. Because they need a job to survive but they’ve been laid off in staggering numbers.

Actions:

  • Directly give people money
  • Give people food and water
  • Give people shelter
  • Get people jobs
  • Transition to a UBI or UBS (although be careful)

Q2. Why have people been laid off?

A2. Due to the lockdowns and economic downturn because of the Covid19 pandemic.

Actions:

  • Stop or reduce the effects of the pandemic as much as possible
  • Help invest in vaccines (Bill and Melinda Gates foundation did this well)
  • Help reduce vaccine red-tape (this has been done)
  • Help reduce the misinformation around vaccines so enough people get vaccinated we’ll have herd immunity (there’s a whole other issue about algorithms prioritising screen time causing issues with our collective sense making here)

Q2. Why can’t the government or the markets save us from this situation?

A2. Because there’s political and economic issues to simply printing more money.

Actually the governments of the world have been printing lots of money but there’s core system issues, from increased inflation that will cause the poor to have even less money, to technological unemployment. All while because of the profit motive mixed with our values system we are creating more and more externalities causing grave environmental damage and human suffering.

Q5. Why aren’t we changing to a different system?
A5. We should be working towards transitioning to an abundance centered society (also known as post-scarcity), but there’s not nearly enough resources devoted to it.

There’s a number of groups working towards a post-scarcity society. They include The Zeitgeist Movement, The Venus Project, Game B, Social Ecology / Communalism and more.

I define a post-scarcity / abundance centered society as one where at least the necessities of life are free, for everyone on the planet.

There’s many aspects to a potential abundance centered society. Most require a full systems redesign using Cradle to Cradle materials flow so things are designed to last, or to be bio-nutritional (good for the environment). They’ll use access abundance, aim to have better collective sense and decision making. Obviously everything from automated production to the scientific method are all important.

For a better overview of both the issues of our current system and an overview of a new one I highly recommend Zeitgeist Moving Forward.

I’ve also got a presentation about transitioning to a post-scarcity society.

Cheers!

Michael Kubler

michael@zeitgeist-info.com

2020 – A year in review

Here’s a general look at my year of 2020.

The year that started off with epic bushfires in Australia, the Taal volcano in Philippines erupting and will be known as the year of Covid19. There was the massive explosion in Beirut, even brain eating amoebae in a part of the US water supply, the Arecibo observatory collapsed and a whole lot of political shenanigans in USA. Although at least Trump is on his way out, even if he’s flailing about as he goes. Thankfully the year hasn’t ended with an Alien invasion or something crazy.

This year. It’s been an interesting one. Certainly a start of the decade worth remembering.

For me, the main highlight of the year is that it’s when Xavier Jayden Cabahug Kubler was born, on the 15th of January. My first born Son and the start of me becoming a father and having a family with Jen.

It’s the year I declared my plan for helping transition to a Post-Scarcity society, with the next 5 years being prep and then 30+ years of transition.

It’s the year I started working on the Gather Together video analysis system that’s aimed at helping Stock Footage content creators keyword their content.

I also started but haven’t been able to work much on an app with my Sister for helping parents dealing with their kids medical needs (more to come later).

Unlike most people, the lockdowns barely affected my work. I have been working remotely for a few years now and it’s been working well. I’m still learning about myself and developing new coping mechanisms or better procedures.

Today, New Years Eve the 31st of Dec 2020 is the day when tomorrow is next year.

It’s been:

  • 933 days since Jen and I first met (2.6 years).
  • 676 days since I moved to the Philippines and started living with Jen. Also more than that since I’ve seen my family except via video call.
  • 351 days since Xavier was born (making him 50.1 weeks or 11.5 months old).


Family:

  1. The big obvious thing is that Xavier was born.
  2. I’m still living in Pasig, Metro Manila, Philippines with Jen, Xavier, our pug dog Maui, and our fish.
  3. It’s been amazing seeing Xavier grow up. He can now walk some steps unassisted and nearly says mumma and dadi. He’s very feisty and I hope he’ll grow up to be geisty.

Work:

  • I’m still doing web development for Mozzler. I’ve tracked at least 1,047 hours of work and I love working with Chris Were.
  • Worked on Viterra, Drakes, Drivible and some other projects. Mostly Yii2 with our own framework extras, but also some NodeJs. I’ve got 745 contributions listed on GitHub
Github Contributions
  • I started work on my own video analysis platform. The core works but I’m still not yet ready to release it. I’ve worked about 128 hrs on it this year, mostly some Sunday’s since July.
  • I have been working on stock footage creation in the mean time as well. Currently have made USD$503.73 in total from BlackBox submitted stock footage.
  • Been meaning to do some work on a mobile app with my Sister but other work and family commitments kept me hectic.

The Zeitgeist Movement (activism):

  • Helped put together the Aug Newsletter
  • Helped Cliff with the global Zday 2020 event, despite me being in the Philippines and it was in Iceland.
  • Did some work on the TZM global website.
  • Helped remove someone from a position of power whose actions were both bad for himself and the community.

Gaming ~200hrs worth:

  • Destiny 2 – I played about 46.4hrs of this, but most of it was in 2019 with only a tiny bit at the start of 2020.
  • Doom – 25.6hrs played back in January whilst waiting for Xavier to be born.
  • Borderlands 3 – 12hrs it didn’t really draw me in
  • MarZ: Tactical Base Defence – 2.5hrs of a turret defence game that wasn’t as mindlessly fun as I was looking for.
  • Reprisal Universe – 10.7hrs played of the spiritual successor to the first Populous
  • Doom Eternal – 33hrs in July.
  • Terraria – 3.3hrs according to Steam, but it certainly feels like I played it for a week and enjoyed it, but wanted to play Minecraft with RTX enabled instead (but that wasn’t released to the public yet, just teased)
  • Serious Sam 4 – 23.4 hours of fun. I loved this mindless game. Played in October
  • Star Wars Republic Commando – 6.4hrs An old game I was playing a little until Cyberpunk.
  • Cyberpunk 2077 – 75.2hrs played since it’s release, 41.8 of those hours in the last 2 weeks. It’s a great, immersive game similar to GTA but in the future and has Keanu Reeves. I’m aiming for 100hrs with this.

Health:

  • 1,758,072 Steps for the year. 1,499.7Km walked/run for an average of 4,817 average steps a day (less than I’d like the aim is 5,200).
  • Lots of running. Some great trips a decent number of 12km runs and even a 20km run. A total of 524km run, with an average of 10km a week.
  • Got a skin rash, likely triggered by my latex allergy. Also a ringworm like fungal infection.
  • Very little in the way of colds and flu this year. I remember feeling a little under the weather for half a day. The face masks and now face shields are helping.
  • Had some issues with emotional

Articles and videos:

I’ve posted 8 posts on my kublermdk.com blog if you include this, plus some posts to Medium, most I cross posted.

The main ones being:

Travel and experiences:

Obviously it’s hard to get out with Covid19 and a baby boy so we only did 1 main trip.

A trip to Amami beach resort in Puerto Galera for my Birthday. I still need to write up info about travelling in a time of Covid. Thankfully it was a great trip and Jen organised videos from my family, which made me cry.

Deaths:

  • Mario Matiev – He was a friend and fellow TZM member. We’d been Skyping regularly for over a year when he finally died of kidney failure after being on dialysis for a rather long time. I miss Mario and he was working on some great stuff.
  • Paul Parker-Benton – Technically my older brother-in-law but someone I only met a handful of times. Still, his passing affected the family and it was weird seeing a funeral via live stream.

Items and things:

They say that for a happy life you should focus on experiences not materialistic goods. But when you can’t travel much and you live and work at home in a one bedroom unit shared with your partner, baby boy, dog and fish, sometimes things can be useful.

  • New Huawei GT2 Pro smartwatch. This was a great Xmas present from Jen and great timing as my Garmin Forerunner 235 died between Xmas and New Years Eve. The watch doesn’t just track my heart rate but also stress (based on Heart rate variability I think?), it has nicer sleep graphs, even if it’s opinion is that we should be sleeping from 10pm until 6am. The graphical touch display is awesome, the watch has it’s own speaker and I can play music on it or even use it as a mic source for calls (I think, haven’t tried this and I don’t think it works with Whats App, etc..). The watch has 2GB of space, even has a calculator app, torch mode, Air pressure, Compass, etc.. Jen had to sell off her old phone and use all her money from selling bags and things, but it is a very elegant and nice watch. It helps that I got her a Garmin Fenix 5 smartwatch.
  • DJI OM4 smart phone Gimbal – I got this a few months ago for capturing better stock footage but also for filming family stuff with Xavier, etc..
  • eBike – Jen organised a decent enough eBike also for my xmas present
  • Live Broadcast microphone setup – Jen got me a full microphone with pop filter, sound mixing deck with sound board and the ability to make people’s voices sound like chipmunks amongst other things. I hope to do some good audio recordings with this one.
  • Washing Machine – After the full lock down started just after we got Xavier home we couldn’t go to the nearby laundromat and had to wash our clothes by hand. So getting a Washing Machine was one of the first big things we did after the Covid19 hammer was lifted (and replaced with the dance).
  • 10TB HDD – I got a WD RED 10 Terabyte hard drive to keep some of my footage safe. Along with the 8TB drive, some 4TB drives and the files I upload to BackBlaze, Degoo, Dropbox and Amazon S3 to ensure I have copies of my important data.
  • A big 72L fish tank – We were given a big aquarium for free a month or so ago as the people who had it moved out and left it behind. It took us a while to get the bacteria working properly and lost some fish in the process as we were too eager to buy more. But it’s working well now.

Others:

Was affected by 2 Typhoons barely a week apart. One when we were at Amami beach, the other when we were back home but there was major flooding and the power went out both times. We somewhat slept through plenty of other typhoons but those two were more than just a storm to keep inside for.

During the strict lockdown time we had our toilet cistern spring a leak and our bathroom sink kept trying to fall off. It was nearly a month of dealing with it before stores opened up that contained what we needed to fix it.

I still am worried about the sink falling forward, but really want to take the toilet working for granted as having to bucket water into it just isn’t fun.

Hopefully 2021 will be an interesting but less onerous year. As I posted on Facebook, I hope it’ll be less like a bad sequel and more like Endgame was to Infinity War, a great end to a 2 part series. Although with the way US politics is going it might be more like Harry Potter and the Deathly Hallows.


Some videos of the year:

Google AutoML Prediction with a Google Cloud Storage source

As per the Gist https://gist.github.com/kublermdk/0b8c1f6173e5b121e5aee303160fa3f3

<?php

// --------------------------------------------------
//   Example Google Cloud AutoML Prediction
// --------------------------------------------------
// @author Michael Kubler
// @date 2020-10-07th
// This is a cut down gist of what you need to
// make a Google Cloud AutoML (Auto Machine Learning)
// prediction request, based off an already uploaded
// file in Google Cloud Storage (GCS).
//
// The main point is that the payload to be provided
// needs to include a Document
// the Document needs to have an DocumentInputConfig
// The DocumentInputConfig needs a GcsSource
//
// Those things took longer than they should have to
// find and work out how to use.
// The Documentation is auto-generated and hard to
// understand.
// Semi-Useful links:
// https://cloud.google.com/vision/automl/docs/predict
// https://googleapis.github.io/google-cloud-php/#/docs/google-cloud/v0.141.0/automl/v1/predictionserviceclient
// https://cloud.google.com/natural-language/automl/docs/tutorial#tutorial-vision-predict-nodejs

use Google\Cloud\AutoMl\V1\PredictionServiceClient;
use Google\Cloud\AutoMl\V1\AnnotationPayload;
use Google\Cloud\AutoMl\V1\Document;
use Google\Cloud\AutoMl\V1\DocumentInputConfig;
use Google\Cloud\AutoMl\V1\ExamplePayload;
use Google\Cloud\AutoMl\V1\GcsSource;
use yii\helpers\VarDumper;

// -- Things to change
$autoMlProject = '186655544321'; // The ProjectId - Set this to your own
$autoMlLocation = 'us-central1'; // For AutoML this is likely to be the location
$autoMlModelId = 'TEN15667778886635554442'; // The modelId - Set this to your own
$autoMlCredentialsLocation = __DIR__ . '/google-service-account.json'; // Set this to where ever you set your auth credentials file
$gsFilePath = 'gs://<bucket>/filePath.pdf'; // Obviously set this to your file location in Google Cloud Storage

// -- General setup
putenv('GOOGLE_APPLICATION_CREDENTIALS=' . $autoMlCredentialsLocation);
$autoMlPredictionServiceClient = new PredictionServiceClient();
$autoMlPredictionServiceFormattedParent = $autoMlPredictionServiceClient->modelName($autoMlProject, $autoMlLocation, $autoMlModelId);

// -- Setup the request
$pdfGsLocation = (new GcsSource())->setInputUris([$gsFilePath]);
$pdfDocumentConfig = (new DocumentInputConfig())->setGcsSource($pdfGsLocation);
$pdfDocument = (new Document())->setInputConfig($pdfDocumentConfig);
$payload = (new ExamplePayload())->setDocument($pdfDocument);

// -- Make the request (Here we actually do the prediction)
$autoMlFullResponse = $autoMlPredictionServiceClient->predict($autoMlPredictionServiceFormattedParent, $payload);

// --------------------------------------------------
//   Output #1 - All as JSON
// --------------------------------------------------
// You've got a couple of options now, you could return the full set by outputting / returning the serializeToJsonString response
echo $autoMlFullResponse->serializeToJsonString();

// --------------------------------------------------
//   Output #2 - Get just specific fields
// --------------------------------------------------
// Or for this example you might only want the payload[i].displayName and payload[i].textExtraction.textSegment.content
$payload = $autoMlFullResponse->getPayload();
$autoMlProcessedResponse = [];
foreach ($payload->getIterator() as $payloadEntry) {
    /** @var AnnotationPayload $payloadEntry */
    $autoMlProcessedResponse[$payloadEntry->getDisplayName()] = $payloadEntry->getTextExtraction()->getTextSegment()->getContent();
}
echo VarDumper::export($autoMlProcessedResponse); // PHP array format, you'd probably want to JSON encode it instead

// NB: You'll likely want to convert this to a class and provide the $gsFilePath in a method and return the expected response not output it

Reinvigorating TZM

At a meeting last night, Friday the 10th of July 2020 about 15 TZM members had a discussion on Team Speak about trying to reinvigorate the movement.
There was lots of ideas. But a couple of people’s suggestions were on approaches to working out the best option instead of just ways to make TZM great again.Aaron Frost pointed out the need for the Scientific Method and Erykah pointed out how we need to do a post-mortem style review to work out what went well and what didn’t.
Victor tried getting people to sign up to his proposal of doing face to face, street activism and only doing that. Seeing anything else as a distraction that should be shut down.There was a suggestion that we need to go back to the old, more authoritarian organisational structure. A so called “return to the good old times”.But Kees pointed out the Google trend for “Zeitgeist Movement” which is similar to the graph that was in my head, except it drops off much more extremely. It shows that there’s now only 1% or less interest in the movement compared to at the start.
Some suggestions included creating more videos and media and I know one person organising a group working on Podcasts.Personally I think what matters the most is Juuso of Koto Coop who is creating an actual RBE aspiring community which is work towards the actual transition instead of just getting the ideas out.
In terms of core members the movement definitely had a lot of people burn out and do their own thing for the last few years. We also have a habit of burning out at least one good member when they do a global Zday event. Casey, Franky and likely this year it’ll be Cliff.
Late 2018 is when it feels like the movement was at it’s most fragile and very nearly disappeared. But thanks to people like Juuso, Mark, Cliff and the Discord community we managed to keep it going. I personally credit the tenacity of Mark for keeping the global meetings going and making them easy, open and transparent.We also used the opportunity to re-organise the movement. There used to be a pyramid hierarchy of communication. Local chapters would report to the national coordinators who’d report to the Global Chapters Administration. There’s now no longer a central gate keeper group like there used to be.  It’s a lot more distributed. Partly based on my Reorganisation doc, plus some other peoples ideas.
Different people have different areas of responsibility and in most cases there’s different groups who help run different projects.
Myself I’m the main person with access to www.thezeitgeistmovement.com and as such website updates are something I prioritise. I can also post on Facebook so I’m usually involved in things like the Zday events which need updates and promotion.
There is a team responsible for moderating the Discord server, another group who deals with the main Facebook page but again others who deal with a lot of the other Facebook groups. There’s Telegram, Team Speak and more.
It seems that especially during the global Covid19 pandemic there’s been an increase in people interested in the Zeitgeist Movement.That’s not surprising given the fact we are being forced to go into a form of economic hibernation and because the existing capitalist monetary system doesn’t support that there’s some room for change. Something we’ve been wanting for a decade.
So with people interested in making TZM great again there’s a few things to consider:Firstly there’s a question of if reviving TZM is a good idea

As the Zeitgeist Movement is about a systems perspective take on transitioning to a Post-Scarcity society (e.g NL/RBE) using the Scientific Method, Sustainability, Access Abundance, Automation and Technology and the like.
Yes. I think the movement has an important place. It has the potential for far greater long term positive change than Occupy, Extinction Rebellion, Oxfam or even the Red Cross. If you’ve watched Zeitgeist Moving Forward and understand the train of thought then you’ll understand why.
Secondly is how would you go about it.My proposal would be along the lines of “We need to reinvigorate the movement in order to help transition to a post-scarcity society. To do that we need to know why more people aren’t more active in the moment, work out what what we are missing and what is the most effective activities we can do

Just from the meeting alone we have some ideas of what might draw people in, like:

  • Face to Face street activism – Although not during the current Covid19 issues
  • More online media. Videos, podcasts and the like – There’s a great Podcast team that’s being created.
  • Online community spaces – We have FB, Discord, Telegram and the like, so this is mostly taken care of. Although the website needs a bit of a content overhaul.
  • More clear messaging – There was a project by Cliff creating new explanation videos which started this but has taken a backseat whilst he organises Zday.
  • More consistency – The example given was of all chapters having the same making scheme. Personally it sounded a bit OCD and I think we need to have locally appropriate diversification.
  • A change in org structure – Note that this has already been done, just not heavily communicated.
  • A change in target audience to be less conspiracy theory based – This is an interesting one and would require most filtering measures which go against the movements ethos of anyone with a good understanding of the concepts being a member. I suspect spinoff groups could add their own filtering in better ways.
  • A practical, physical manifestation of a transition. E.g Koto co-op.

I’m sure there’s plenty more ideas.

Obviously which tasks people take on depend on their personality, skills and interest. So there’s no one size fits all approach.

It’s likely we need to do a review of historical events until now and if possible interviews with people who are no longer members and people who don’t know about the movement and try working out some experiments to see what is actually effective.

One thing PJ said about the movement is that it’s excitement based.

Being an active TZM member I’ve of course talked to many people about the movement and ideas and something I get very consistently is people saying “So you’ve been around for 10 years and what have you achieved?”

 As TZM is about promoting the ideas of the NL/RBE we explain how we’ve reached lots of people. Millions of views on the main videos. Large amounts of media content, lots of chapters, events and activism.

But people want to know what steps we’ve made towards the transition.

As per my Price of Zero transition talk. That’s where the Crossing the Chasm marketing information comes in useful to understanding why so many people are asking this question. Only a very small percentage of people are the innovators and early adopters of new concepts. Most are practical minded. They will join an RBE aspiring community to get things done they can’t under a capitalist society but they won’t go building the initial prototypes.

So based on my current understandings I think one of the most powerful things we can do is help people who already know the concepts to know we are making steps towards the transition. I’m working on a 5+25 year transition plan of my own.

But in the mean time there’s Koto Coop which is just starting, Kadagya in Peru and a handful of proposals which need resources to get off the ground.

Still, doing the experiments is important and I think we need to work on the metrics which define success.

It is about more members in some specific community (e.g Facebook, or Discord)? That’s an indicator of marketing not actual transition progress.

Is it about the Google Trend line of Interest Over Time going up? This would likely indicate more people interested in knowing about the movement. But partly the virality which helped spawned the initial interest was shaped by the cultural environment which has changed.

Is it about how many political policies are altered, or political parties voted into power? Not likely as that’s not systemic change.

Ideally we’d have the Zeitgeist Survey Project to have a better handle on the actual change in cultural zeitgeist and also some metrics for tracking systemic change. But that’s another project to work on and needs lots of help to get started.


KotoCoop: https://kotocoop.org/about/model/

Google Trends: https://trends.google.com/trends/explore?date=all&q=zeitgeist%20movement

–Michael Kubler
Email: michael@zeitgeist-info.com
FB: @kublermdk

Advanced Filtering with MongoDB Aggregation Pipelines

This is a repost from https://medium.com/@kublermdk/advanced-filtering-with-mongodb-aggregation-pipelines-5ee7a8798746 although go read it there so I can get the Medium $$, because it reads better.


For a project Chris Were and I have been working on we discovered a great way of doing advanced queries using MongoDB’s Aggregation pipelines with the use of the $merge operator in order to filter down to a specific set of customers based on their demographics and purchasing behaviour.

For those that don’t know MongoDB’s aggregation pipeline is much more sophisticated and at least for me, also more intuitive than filtering using Map-Reduce.

The project is part of the APIs and admin control panel backend that powers a mobile app for customers who are shopping.

We know things like the customers age, gender, and state.

We save some of the customer’s transaction data, including the storeId, an array of products and an array of product categories. Although only the last few months worth.

We have an admin control panel where the managers can create a set of filters to select a specific customer group, they can then then send push notifications to those customers, assign them coupons or send them surveys.

Except for the number of days ago, the entries allow for multiple selections. e.g You can select all age ranges, just one, or a couple of them.

Although it’d take you a really long time in the UI to select almost all of the products or even the categories. Hence we have both have and haven’t purchased / shopped at versions.

Example filters:

Age: 0–20, 21–30, 31–40, 41–50, 51–60, 60+

Gender: Male, Female, Other

State: South Australia, Victoria, Queensland, New South Wales, Western Australia, ACT, NT

Have Purchased Products [X,Y, …] in the last Z days

Haven’t Purchased Products [X,Y, …] in the last Z days

Have Purchased from Product Categories [X,Y, …] in the last Z days

Haven’t Purchased from Product Categories [X,Y, …] in the last Z days

Have Shopped at Store(s) [X,Y, …] in the last Z days

Haven’t Shopped at Store(s) [X,Y, …] in the last Z days

They needed the system to be flexible so there’s also include and exclude versions of the filters.
Importantly the Include filters are AND’d together whilst the Exclude filters are OR’d together. So you can get very specific with the includes whilst applying some broad exclude filters.

An example might be selecting people who have purchased toilet paper and alcohol hand sanitiser in the last 7 days but exclude all people aged 60+ and all people who’ve purchased kitty litter. A notification can then be sent about how there’s a new pandemic preparedness set which is now in stock, or how the stores are being regularly disinfected during the Covid19 pandemic.

Another option could be to target people in South Australia who are 30 yrs old or under and have purchased from the Deli – Vegan meats product category in the last 10 days, but exclude those who’ve purchased the new Vegan burger. Then they can be given a 2 for 1 voucher for the new burger.

With many thousands of products, hundreds of categories and a few dozen stores there’s a reasonable amount to search through. We also don’t know how many customers there will be as the system hasn’t been launched yet.

But the system has to be fairly fast as we sometimes need to queue up customer push notifications whilst processing the HTTP requests from the 3rd party sending us the transaction information.

The important parts of the data structure looks like this:

What we needed after after filtering all the customers and transactions is a list of Customer IDs. We can then feed those into a variety of systems, like the one for sending push notifications, or selecting which people get coupons.

A main limitation was that whilst the database servers were quite powerful, the web servers weren’t. My initial thoughts were to process an bunch of aggregation pipelines, get a list of CustomerID’s and do the merging and processing in PHP, but when there’s potentially 100k+ customers and transactions, Chris pushed to go harder on the database. Thankfully MongoDB is powerful and flexible enough to do what we wanted.

In the latest v4.2 version of MongoDB there’s now a $merge aggregation pipeline which can output documents to a collection and has some advanced controls about what to do when matching, unlike the $out operator.

I worked out that we can do two types of queries. A “Haveselect of all those who should stay and a “Have Not” select of all those who should be removed.

For a “HAVE” Select we output the customerId’s into a merged results collection for the customer group with an extra {selected: true} field and bulk delete those without the selected field, then and bulk-update and removed the selected:true field.

For the Have Not’s we select all the customerId’s of those we don’t want, set {excluded:true} and bulk delete those with the field.

Example of Include and Exclude filters

This is an example of the UI for setting the filters. The approximate customer’s is based upon some randomly created data used for load testing… In this instance 17k customers and 340k transactions.

The UI creates a set of filters with the values of things like the productId’s, but the PHP backend, using the Yii2 framework, Mozzler base and some custom code, does some parsing of things like ages. e.g From the string “0–20” to the current unix “time()” to “time() -20 years”. Similar changes are done on the backend to convert something like 60 days ago into a unix timestamp relative to now.

I was going to do an aggregation pipeline for each filter. However if there’s 10 filters that could be a lot of work. MongoDB seems to be better at having somewhat complicated $match queries (with some good indexes) but not so good at running as many aggregations.

Chris then suggested we merge the aggregations and I realised that doing it the following way actually works out perfectly and we only need a max of 4 aggregations. I’m sure if the Exclude filters weren’t considered a logical OR or the Include’s were consider a logical AND of the filters then things would be different.

Aggregation pipelines

1. Customer Select (Have):
Include filters for a Customer’s Age, Gender and/or State

2. Customer Exclude (Have Not):
Exclude filters for a Customer’s Age, Gender and/or State

3. Transaction Select (Have):
Include filters for Have Purchased Products, Categories and/or at Stores
Exclude filters for Haven’t Purchased Products, Categories and/or at Stores

4. Transaction Exclude (Have Not)
Include
filters for Haven’t Purchased Products, Categories and/or at Stores
Exclude filters for Have Purchased Products, Categories and/or at Stores

From the admin control panel UI we’ve grouped the filters into the Include or Exclude and have an array of them.
Because things like 7 days ago needs to be converted into unix timestamp based on the current time, we need to update the aggregations dynamically, hence using MongoDB Views wasn’t really possible.

On the backend I wrote a Customer Group manager system for grouping the filters into the different categories and merging them together.
Whilst the actual queries we did were a bit more complicated than shown below because we aren’t saving the state as a string but a postcode so have a bunch of ranges for them, what we do is very similar to the example aggregation below, based on the filters in the UI screenshot:

The aggregations in the above Gist should have enough comments to explain the actual steps in detail. But it’s expected you’ve used MongoDB’s aggregations to have some idea of what’s going on.

Some points we discovered:

  • The $merge filter lets us put all the data into a merged collection and we can iterate over the results using a cursor, use the collection count and other things to make it both easy and very scalable. The merged collections are effectively free caching.
  • It was very powerful to always be doing a $match (select) query first. It’s fast with the right indexes and powerful with the excluded or selected: true and doing a bulk delete / update.
  • Merging into the Have / Have Not filtersets on the Customer and Transaction models mean there’s a maximum of 4 pipelines that will be run. Although if the Exclude filters were an AND not OR between them, then this might not be the case.
  • An edge case is that we have to add in all customerId’s as a first pass if there wasn’t a Customer Have pipeline so that customers who don’t have any transactions could still be returned, or so that if there’s no filters then it selects everyone.
  • I also developed some tweaks which let us run a set of filters on just a single customer or small selection of customers to know if they are part of a customer group. This is especially used when a new transaction comes in. We point to a different (temporary) customerGroup collection in that case. Obviously querying against a small set of customerIDs makes things much faster.

The end results are good. I get around 600ms when querying against 20k customers and 180k transactions with the basic Vagrant VM on my laptop. Although that is random data I generated just for load testing.

We are still waiting to see what this will be like in production.

Let me know if something doesn’t make sense or if you want more information.

Via Negativa

Via Negativa translates to “by removal.” Taleb argues that a lot of problems can be solved by removing things, and not by adding more. In decision making, if you have to come up with more than one reason to do something, it’s probably because you’re just trying to convince yourself to do it. Decisions that are robust to errors don’t need more than one good reason. You can observe the beneficial effects of Via Negativa effects in a vast number of fields from Medicine and Diet to Wealth.

Copied from: https://anantja.in/antifragile-things-that-gain-from-disorder/

This mental model wasn’t in the main list of mental models I often look at https://fs.blog/mental-models/ but it’s one I’ve come across before and wanted to point out to people so thought I’d post it here on it’s own.

In this case, I’m thinking about using Via Negativa for removing stupid people from a group.