doot-doot-doot boot hey you’re alive
what’s going on guys hangin at my commission it’s going on with you man
yeah yeah it’s been a while this is our very first official conversation which
is exciting so we’re gonna be talking AWS DevOps e-types stuff
Andrew how did engineer at sight words Magento enterprise solutions partner and
you do a lot of in-house DevOps stuff for your clients and then Winston Nolan
in the right box at least for me works for a very large flash-sale merchants
out in South Africa and in a chatroom that I’m a frequent with some of the
smartest people that I know in the Magento community he’s like the DevOps
God so you guys got it thank you for having me on yeah absolutely man
absolutely these designs getting this fun I’ve been wanting to do this for for
a little while so I don’t know where to start
I mean it might be interesting I mean I’m gonna basically just you know you
guys are most going to be talking most of the time I’m gonna I’m gonna try not
to look too stupid is basically my goal isn’t this whole thing try not to say
too many stupid things all at one time but yeah I mean I don’t know might be
interesting Winston for you to talk a little bit about some of your
infrastructure it’ll be cool to get kind of different perspectives between
merchants side and agency side okay you know I you know so so Winston – take it
away man tell us a little bit about well I also also learned about your
background and how you got into this racket and all that kind stuff okay all
right so so I’m Winston Nolan I am an operator in the IRC Magento Channel I
started with Magento way back in 2007 I believe 0.71 beta and back yeah back
then there was only what was it called OS Commerce and
so me and my I’ve got an identical twin brother and me and my brother we bought
a CD shop and we were selling hoodies and clothing and mp3 and wav files over
the Internet I didn’t even know this I don’t even realize all this yes yeah and
I had a day job back in as a Systems Administrator and so in the evenings I
would back all my orders take all the hoodies and the t-shirts and all that
kind of stuff back them nicely and then in the morning I take a break from work
at about 10 o’clock and I’d go and ship this stuff via the national postal
service and it was crazy man because the stuff we just never arrived and then the
guys would complain and then two days after you ship another parcel the
original parcel would arrive and so there was some complexities there but
you know magenta was just from the get-go it had such a good feel to it it
was sufficiently complex it could give you a lot of features and I’ve carried
all along that vine I started building bigger and bigger ecommerce sites
started doing freelancing and then I got snapped up by media 24 which is a big
media house in South Africa and for them I bought a what a few stores and then
you know this is how I’ve travelled my interest I will say this DevOps thing
comes from when I joined run my cell about four years ago this is the company
that I worked for in the day I’m the head of tech there and back when I
started I was the only developer the only permanent developer and we just
inherited the site from an agency was a science so we can you see anything about
this sort of the scale of a runway sale just so people have an idea of the
levels of yes so on our busy days we go to we go from about 100 active cold
currents to 4,000 concurrence in five minutes there we go
from 5,000 active currents to 10,000 active cold currents in another five
minutes um so our site go from 500 to 10,000 we do quite a few orders on these
days and so we are to just scale Magento make sure that our infrastructure can
scale and this is really where the beauty of AWS is what we host on came
through and you know I suppose each each company does things differently we don’t
have a permanent in-house DevOps person as we only have a DevOps person for a
couple of hours a month so our whole infrastructure was built around the
notion of infrastructure as code configuration in code and scaling this
thing as fast and as high as we can so for example our staging can go to equal
to production or towards 10 production in about a half an hour this is if I
have to scale it scan it manually typically before we have these very busy
guys like Black Friday I will scale you know a couple of days before and all
that kind of stuff and yeah that’s I guess it in a nutshell that’s intense
that’s intense sorry so I interrupted you or you’re in the middle of giving
your whole your whole spiel and I just want was curious to get some rough
numbers on the scale of runway sale but no not at all it’s a beautiful journey
man I I love this I love I love ecommerce on cloud systems and I love
this whole automation automation vibe and you know through that by adding
limited resources available with your fluent are to do things a bit smarter
and and all that kind of stuff so it’s really interesting and Andrew a little
bit about your background and and what you do keeps you busy
too much so my background basically I started I used to own a small company
but that was quite something years ago and as part of that small company we did
a couple of different things we ran training people like fitness training
people we had like a little shot that sold fitness gear so basically I got
into Magento because it ran on cPanel and didn’t break every 10 seconds and
that’s kind of my entry into this this market right there’s Rick Knoll so that
was pretty cool we did that for maybe two years and then moved on to a startup
which was an experience that involved building a selling platform on top of
Magento where merchants could list their products and then would resell on
through magenta and that was my first kind of play with AWS we used a service
called elastic beanstalk which was certainly interesting it’s like AWS as
Heroku that’s pretty straightforward that’s pretty cool
but unfortunately the the startup didn’t live to see success so moved on to a
steel company in Melbourne called pontus they’re like an agency much like
cycloids and it was from there from a couple of guys there a guy called Lloyd
Haslett and a guy called Matthew handle but I sort of that I learnt the trade of
sue sending those guys super super smart and now extremely generous with the
knowledge so work there for a couple years doing everything basically and
then I was talking to David one day and because then you David 3 Twitter and
he’s like dad to come and work in Germany and I’m like Germany seems
pretty cool so got the job off Twitter and moved to Germany and been here ever
since I don’t work here in any capacity as a
develops person we’re much like Winston we don’t really have a dedicated DevOps
team the way works here is it’s like we have a project team at one person in
that project team will will manage decider for the hosting stuff so I tend
to take all of the hosting type tasks yeah that’s basically it gotcha gotcha
yeah and I um it’s funny every time I talk I can see your face light up with
orange because your screen is probably showing yeah I know and I thought you
know you um seem to be a big proponent of like agencies having I mean you don’t
call yourself like a DevOps person but it seems to be a decent chunk of what
you do or what you’re what you’re thinking about what you’re interested in
and I know we’ve had some chats about that like that that whole strategy
around you know agency having kind of an in-house person that sets up all the AWS
infrastructure for clients and things like that and so the problem that we had
so first I started this this journey at fountas as I said who did all of their
own hosting so that wasn’t necessarily a choice it’s just kind of like alright
here it is is the hosting learn Linux good luck once we got to sight words the
problem that we are having we used a couple of different hosting partners and
the problem that we were having is there was difficulties communicating the exact
nature of specific bugs or getting particular things resolved or
limitations the environment and what we’d find is that we’d start working
around the limitations of the hosting provider or that would be we’d be doing
kind of crazy things to make everything work or get the information that we
needed so about a year ago probably a year and a half ago I got licensed to a
client up completely on their own independent of ahead of us because they
were they were concerned about getting locked in they’ve been locked into their
previous provider and the provider was holding them hostage
so you said yeah great for sure and have your own AWS account this as always
yours we can’t take it away from you you have rights
great catania and that turned into quite a success we built that the
infrastructure is Krone approach and we built continuous delivery and continuous
integration and that system is now deploys regularly and it’s really
reliable and really relaxed as compared to a previous experience where
deployments were powered by prayer basically
a prayer right yeah yeah giving clients the the control there is is um it’s cool
because yeah as a client if your depend on a hosting company and things don’t
aren’t going so well that could be a pretty nerve-wracking situation to be in
so well cool so so like what what are what are some of the like Winston what
are what tell us a little bit about like your setup this is where my eyes are
gonna glaze over a little bit but I’m sure everybody that’s you this is where
you guys are gonna just start doing your own thing here but like a little bit
about your setup and maybe some things that other you know Magento stores might
be able to to think about as far as your infrastructure goes okay so okay so
first of all we host entirely in Ireland and that is the closest AWS presents to
us so straight off the bat we lose about a hundred and seventy hundred and forty
milliseconds Wow office into a farce of our site it’s quite a bit it’s quite a
bit but what we do is you know through various levels of caching we get up
outside to load sub-second you know it’s that it gives that feel of
instant loading okay so so basically our stack looks like we’ve got a varnish
cluster right at the front that you hit that then caches or seems request to the
backends the backends is an auto scalable cluster of appear to be seven
and fpm and nginx web heads we run PHP 7 alongside that we’ve
got Aurora we’ve always got a writer and we’ve got a reader in that Aurora
cluster and then we’ve got separate Redis instances one instance for
sessions alone another instance for cache and we’re gonna you know we’re going to
expand on this that is basically our let’s say that the meat on the bound
that and it’s worked out really well like we
don’t really have a lot of conflict travel I suppose
it’s it’s very different if you’re in an agency because every client has its own
commands its own conflicts that they need to run but we are very lucky that
we could standardize this and so yeah that’s basically it we we have
development instances we separated our site through multiple AWS accounts that
gave us the ability to separate risk and also to give the keys to staging and
development to juniors owners got an automated testing testing framework that
would be bolt on code section and we plug that in with browser stack so we
can test our site it’s automated if we make a PR into into a keyway branch that
will run some tests for us yes so that’s basically basically what we have it’s
beautiful so Andrew do you get that stack set up for a higher bar you know
and I’m infinitely jealous of you Winston’s as Winston said the nature of
working in an agency is that no two clients are exactly the same so it’s
brought some really interesting challenges we have been able to
standardize some approaches between clients and I’ll give you sort of a
release process for one particular client which is becoming our kind of
standard mechanism of doing this in future does that make sense yeah so
basically the way it works is we have we have a ticket that comes in like a story
or whatever and we say can you please make this button blue or red or whatever
they have this so a developer would check out a story branch and they’ll
sort of push that to the origin and that creates a machine so we use bitbucket
pipelines like a SDI CD tool to create the machine and we use ansible which is
a configuration management tool to create the environment and simal handles
all of the cool bits like installing PHP and varnish and Magento and and whatever
else and we we open-source a bunch of the stuff that we worked on so if you’re
curious just just kind of go and have a look then the machine is up and running
and the developer does their work and make sure it works that gets sent off to
to QA where they make sure the developer is not crazy and then it gets an
uncomfortable amount of exam and then it gets released into production so we have
one sort of standard pipeline all the way through we create a machine with one
definition and then we deployed in the production machine with the same
definition so everything’s extremely similar development was an interesting
experience we actually dropped the full stack from development because it was
too complex and we’re finding that the people aren’t able to debug things
effectively when it’s a giant pencil world or giant ansible mess so we we run
local developments in a mixture of docker and just VMs that the developers
manage in that’s very nice how do you many sure database seeding and all of
that engine so basically it’s not a couple exit imagine so we when we run
database changes we either as much as possible we try and build them such that
they’re somewhat backwards compatible so when you actually drop the new version
of the application in it doesn’t panic in freaked here where that’s not
possible we put the site into a maintenance mode and just migrating
stuff there additionally we try and keep the the
deployments and the schema changes as small as possible
so we know kind of exactly what’s happening when that’s happening why it’s
happening and we can put a nice time box around it to make sure that either
nothing breaks or that the downtime that it creates is 30 seconds or 60 seconds
or something so nothing too special no magic MySQL replication or anything but
practically we probably see on average about a 15-second
with each employee and we could fix them but it’s not worth it all right we got a
question from Kevin in the chat you all use code deploy slash code pipeline so so we use bitbucket pipelines for
everything the big buckets a pretty good system and there see ICD tool this is
super well built actually considering it’s only 12 months old so it builds it
stores it in s3 and it deploys an artifact from s3 does that answer the
question Kevin he’ll let us know if that that didn’t answer it Winston what about
what about you guys okay so we also use that bucket
it’s it is like Andrews did a very nice system to use we’ve already got the
testing in a pod light the code is also in a very simple pipeline that just
takes the the package and pushes it up to s3 and what what we’ve done done is
we’ve written a deployment procedure in a system called fabric and what it does
is it it’s as HS into every single machine and does a deploy of that
history but I mean we can deploy to to any number of servers it takes us about
9 to 12 seconds and what we also do that might be a bit strange but it’s just how
we do it is we ship all of our assets with each bolt so we ship JavaScript CSS
all the code everything we do yeah Winston I might ask you as well as I
sort of looking at the next next step for us we want to try and move to
immutable builds so basically in a mutable build for context is where you
take a full image of a machine and you install everything on it and then you
deploy the entire as opposed to deploying just the magenta
application and you replace the machine entirely so my question would be do you
see is so kind of moving to that model or do you think that the trade-offs
aren’t worthwhile okay so that is a very good question I know people that are
doing this with games already so for example whenever a deploy happens you
kick a bash script off that creates all image of that machine that gets pushed
to a launch configuration within Amazon and gets attached to the auto scaling
group and it will come up with that source code the issue I have with that
is we do multiple quick deploys in a day so what we can do is we can we stagger
all of our deploys to go out and we’ll let them each go out like literally like
10 15 minutes off to each other that might cause a little bit of bucking if
you want to do that I am not sure if we’ll go that way if we are gonna go to
docker containers and stuff like that probably we could but then again you
know would we run our admin host on docker probably not so so we’d only run
the front-end cluster on docker and already just with that one statement
this complexity that comes in here and so it’s it’s a very it’s a very it’s a
very cool thing to think about I just have to you know plan it properly but I
mean for the moment I’m much more a fan of my whole boat sitting somewhere that
gets distributed to servers in lightening quick time then what I am of
switching the whole you know the whole image going with yeah that makes sense
so the obligatory question you wouldn’t move to something like Kubb in which you
could handle because I got all excited when you said docker for context I am a
massive docker fan but we don’t run any production systems on it just yet no
client facing production system so much as I would love to get into it I
have not yet okay yeah would you move to something like that alone so so we
haven’t looked at kubernetes just yet I am infinitely jealous the way that you
get to play with that but what I’ll have to see is how does that kubernetes
system handle traffic between it so for example if ink is controllers maybe as a
problem and it starts taking down pods or maybe you you’ve overlooked the
defaults and the defaults start playing up when you get insanely hard traffic
there there is value in that there is value in it it’s a huge part of where
the conversation is these days are we ready for that level of complexity I’m
not sure yeah it doesn’t reduce quite a bit of magic
we’ve got colleagues here going through the course at the moment and what’s the
metric that WTS per minute it’s pretty high we actually got a couple other
questions coveted Isaac asked since your guys are using bitbucket do you use the
other Atlassian tools for planning and take a creation etc when we’ll start
with you Winston okay so yes we do use other Atlassian
products we use confluence quite a bit we’ve got a very huge document store
we’ve documented all of our infrastructure when developers join how
to get started that kind of stuff we document everything even features
specifications all that kind of stuff and we also use Chara for ticket
management nice yeah as a pronoun management that’s a nice no we we like
basically every other agency in Germany I’m sure use JIRA
to plan and manage our projects in addition to bitbucket pipelines but no
other Atlassian tools as far as I can take
okay um yeah by the way sometimes the questions jump around you’re on one
topic and then it jumped to another so we’ll we’ll just jump around a lot and
then another question Kevin do either of you guys have AWS certifications and do
you feel that there is any value to having them in the Magento ecosystem okay I have I’ve got no AWS
certifications I’m busy doing a EE associate course through a cloud guru
you can pick them up for pretty cheap on udemy and that course is actually very
nice to get transferred to a cloud carew you’ve got these awesome videos I think
it’s from Ryan I think the guy’s name is Ryan he is very very skilled explained
stuff very simple to you there’s a lot of value in having ‘iw certifications
there’s a lot of value in having any certifications so my advice is yes
everything is going to be on the cloud sooner or later and iws according to me
is such a big player there that and once you understand the principles of iws you
can apply them very easily to any other cloud system because they basically work
about the same right awesome sorry back to that super deep discussion you guys
were having before I so rudely interrupted or or maybe I mean I’m like
maybe give an overview of like docker and kubernetes some of the different
options in that space like what are some of the high-level differences between
those different things right so I can kind of explain the two different
alternatives which we take one of which is ansible and we use ec2 is that kind
of deployable unit and one of which is containers which is the kind of the
alternative so ec2 is a virtual machine spin it up and such apt-get install
whatever you need it’s pretty straightforward and it’s nice and easy
and there’s heaps of talented developers who are really good at managing it
containers are kind of like a virtualized VM also not really so a
container is something like a tarball that you pull down onto your machine and
then you pretend it’s a VM and run stuff inside so some sort of custom-made dead
package would be another another way of looking at Debian package communities is
interesting kubernetes it’s somewhere between AWS and docker it kind of runs
all of your infrastructure so traditionally where you would think of
services like ElastiCache or aurora you would instead use packages on top of
kubernetes like the high availability Redis or the finest girl clustering
stuff so it lets you take any cloud provider we work across currently for
different cloud providers and treat them as roughly the same which is the the
principle value and for us you can learn one set of tools and reuse that
everywhere the client wants to run your data
Girja nice and true how do you guys manage like credentials and and
configuration versioning within Dhaka and ansible based over nice kubernetes
provides this artifact called config map or a secret there are two different
artifacts ones encrypted at rest that’s that basically the only difference and
they mounted into the the container at runtime so say for example you have an
engine X container and you want to have specific configuration for the nginx
container and you don’t want to pack it into the container itself then you would
run the container with reference to this specific contact map and then could the
beedie’s would map the file that’s in the config map apps actually since nginx
flash confident about so kubernetes provides you with a tool to manage
configuration stuff just like any database nice let’s see
what direction we’re gonna take this now what what are is there any mean you guys
have talked a bit about some of the things your your your you know some of
the cutting-edge stuff you’re thinking about looking at and maybe can’t right
now because of current constraints within your team or within your systems
what are some of the different things you guys are looking at as far as
actually I actually started listening to the AWS podcast a month or so ago I
understand very little of it but it’s just fun to hear them talking about new
services that that come out I think the most recent one they were talking about
a chat bot that they built and they talked about how they used lex for it
and how they handle authentication and just all the different services they
plugged in is kind of interesting but any recent services do you guys stay on
top of all the services that come out and kind of play with the master coming
out Winston you first okay so we try to we we definitely try to what we’re
looking at at the moment is you know redshift for our big data mining
historical reports all that kind of stuff we’ve built up quite a quite a bit
of quality data through all of our orders and our customers and our
interactions and all that kind of stuff and we want to start reporting on that
and we’re also looking at some other stuff like some AI for product
recommendations you know how do we handle stuff like abandoned Court you
know how do i how can we do that automate it in automatic way how could
we do that more intelligently and then something that’s caught my eye was that
that that google ai that that can act as a call center agent my store now is so
terrifying and awesome all at the same time beautiful like the little uh
ah how’s that it inserted it just sounded so natural so natural and I mean
I could just see that fit in completely seamlessly within an e-commerce business
I mean you your other asking about an order or you’re asking about a product
or so there is some exciting stuff that we’re looking at nice yeah
what um what are you guys looking at for if we could dig into the product
recommendation use case a little bit that sounds pretty interesting I know
you mentioned redshift I’m not too familiar but maybe you could flesh that
out a little bit what what are you guys looking at to build out the product
recommendations okay so so basically currently what we have is aw a hit works
you know simple recommended products I believe they call it automated
recommended products and we really want to do a lot more in that space there’s a
lot of it’s a lot of research I think you know we’re we’re these big ecommerce
companies are saying how much of their orders comes from personalized
recommendations to their customers and we haven’t quite fleshed out the
direction we’re going to take there but it’s certainly very very interesting and
it makes complete sense so there’s a big focus for us on optimizing our customer
experience on our site is it easy for a customer to shop
that’s the customer get to the product that he wants to get to once that
product is in his court we’ve recently relaunched a reserve court so once you
once you’ve added the product your court that product is reserved for the next
ten minutes in your court you’ve got the stock assigned to you and you can
checkout and you can imagine with a flat sell side people who write literally
fall all over each other adding this product to court and you know you’d get
a lot of light bounces at the at the chip card once this customer is trying
to check out someone has already bought their item and so we’ve had a lot of
good feedback from that and in spite I approached this a lot easier and it’s
not such a you know not so not so much risk to
literally f this product snatched out of your basket before you can check it off
right I thought Angie was gonna jump in with
something there but so so what are the what are the services that you’re
looking at for the for the product recommendation stuff I mean the the AWS
services we are not we have an identified a service se8 it’s more about
how do we wanna build this how can we build this – you know what is the vision
that we have what is the idea that we can implement how is it gonna play
nicely with magenta data maybe not necessarily coming out of magenta but
coming out of at a different data store how’s it going to play along with stock
availability if for requites a recommendation so it’s more about a
strategy about what we want to vote before we go and choose the services
that will fit in with that gotcha kind of early early stage trying to kind
of sketch out how how it will work and everything nice and you any-any
do you keeping an eye on any of the latest services coming out and some what
the nature of agency is basically we want to we want to build a service
offering super reliable service offering and then make it as cheap as possible
and then roll it out to as many people as possible so we can give merchants the
maximum value for cash for their invest so we don’t tend to do a heap of
greenfield stuff on behalf of the merchants and virtual sorry however we
do this thing called improvement day which is basically we set aside a day a
month and then just kind of Greenfield in something that’s called Sur a bunch
of interesting things have come out we have a section of the company that’s
dedicated to creating those experiences it’s opening earlier about but Google
duplex or Google assistant or things like that so we’re creating interactions
with Magento systems and various other services
beyond that same old same old making things run well run fast brunchy gotcha
nice oh my god that duplex thing was so wild I still can’t wrap my head around
that this is pretty awesome it wasn’t beautiful that yeah I was oh
my god it’s uh whether it’s those things like
those AI things or like also the the robot the Boston Dynamics robot robotic
videos that are just like there’s so awesome and simultaneously terrifying
you know can you believe we’re living in this time right I know how encourage
something the other day where they want to start mining asteroids in 2020 which
is like a year and a half away it’s crazy it is a beautiful crazy time
to be alive that’s gonna be a lot of masterly mining mining asteroids right
did you have to do that yeah just yeah just the whole like reinvigoration of
like space travel is is is crazy I mean it was in the fifties or so when you
know we’re gonna go to the moon we’re gonna do all these things and then
everything just sort of stopped you know until Elon busted on the sea it was like
I made a few bucks from PayPal it’s you know let’s go nuts let’s go burn some
money a question from Oscar what is your strategy to deploy in AWS GCP with
multistage server what’s gzp is that Google Cloud that’s the juicy piece
Google cloud platform but I don’t know where it is in AWS
I guess he’s saying what’s your strategy to deploy with either AWS or G GCP with
multi stage server we don’t have any multi like high
availability stuff so I can’t comment what’s a multi stage server start with
that I would guess Winston he’s talking about canary do you think that’s
reasonable yes maybe like a like a like a docker instance moving from local to
development staging production I guess maybe Oscar can conclude is not clear
enough for us to answer I’ve got a question for you man yeah man what what
do I mean obviously you guys are a big Asian city and you you must have a lot
of clients but how do you approach Black Friday as an agency because obviously
you must get a lots of client requests we want to test our site we want to make
sure we really want to upload prices that’s a big one at plov o’clock we want
to change 3,000 product prices we want to invalidate caches we want to have
indexes to run and these process must be live by 12 how do you guys handle that
just to provide some cultural context it’s not as big a deal in Germany just
yet we’re only just starting to see the
agents of the Black Friday craze that start to creep at the edges we
essentially slept through it this year now on I won’t complain no one did
anything the last thing that you probably should know the context is so I
would just traditionally been very business-to-business rather than
business to customer which means we do like lots of high-value transactions and
to integrate with the ERP s and things but high traffic loads are only a
problem in a couple of cases yeah okay okay and for that stuff we just ignore
it if it falls either we declare an incident free get Gotti got a uh
actually uh Winston we should we gotta get you to talk Black Friday stuff how
how is that is that pretty wild out there
where you guys are or is Black Friday not as not as big okay so Black Friday
by us is very big it is it is it is but for the biggest day that we have off the
here this this is whole week leading up to Black Friday and where we take
advantage of the whole week so it’s not only back Friday
it’s Monday to Friday and also the next Monday is called Cyber Monday I don’t
know if you guys have heard about it though so bye-bye us it is it’s very big
and so we are – so probably the biggest the biggest thing for us at the moment
is of course scale and handling all of the traffic and all that kind of stuff
but this some other nuances it goes with it so for example we’ll have a lot of
stock that we’d have to sell on that day you know we would just have a lot of
stuff that we have to Sibel so you can imagine an incident it really eats at
your bottom line on that date really bites you many other thing is like we
would have at a minimum 20,000 product prices that would have to change and so
we haven’t yet built a facility to to schedule those prices and execute them
at a table so with a flip of a switch this price is smaller they say a hundred
dollars and it’s now $80 but what we do what we did do is we built a price
importer that will import promises and clear caches and sate index values and
all that kind of stuff and I think when lost we ran it
Black Friday we did 20,000 products in about was 18 10 15 minutes and all of
those prices were live on the site throughout the catalog listing on the
product view pages on the cart pages as well as the checkout so we’ve done a
fair a fair amount of work in that area and Black Friday for us is whether it’s
always a big thing like we we scale our infrastructure we scale the
database because radars it’s an in-memory alien memory cache store will
terminate Raiders and span up a bigger instance will change our config settings
will have a bigger spec web server will have a bigger spec varnish cache server
and will will run that you know for those busy days and will turn it down so
we’ll tear that infrastructure down might be over a weekend save a bit of
cost there and then bring it up again on the Sunday night for for the next week
it’s pretty interesting and then we’re all at the office at 4 o’clock in the
morning and we’ve got a pretty nice culture so it will have a whole room
just for us and they’ll be like motivational pictures against the wall
that’s like bowls of chocolates and sweets and biltong and coffee and and
all this kind of stuff so the company does take care of us and then what we’ll
do is like mostly everyone will sit with popcorn in front of analytics screen
just watch the slight fly because it you know you know us it’s so big that even
the payment gateways fall over and 3d secure you’ll have problems with 3d
secure so people will start to use different payment methods to to get
there get that what is through very interesting so Winston I would be
curious what architectural changes have you made some magenta itself to make
that possible like I know historically we ran into
orders reserving stock we would end up with lock tables and accurate of things
I’d be I’d be curious to see what changes he made there okay so this
Magento has been coming along for I would say four years now where we’ve
just continually optimized magenta we’ve turned off login with disabled features
we have rewritten sequel queries we have rewritten mechanisms to
if for example we don’t use any catalog price rules
so we’ve disabled all of those indexes on the checkout
we’ve got mechanism so far we move orders from the database into the auras
grid we’ve just done a lot and we continuously do a lot so if you look at
our new relic we’ve got an average PHP response time of a hundred and twenty
five milliseconds and so we’ve both our own load testing framework as well where
we generate load on this site and we make sure that you know why why is this
wise to select so heavy let’s go and call that let’s talk at
this one transaction see if we can make it a bit lighter so we’ve done a lot of
work this in fact so much I can’t even not even remember everything just to
stay up and thankfully we have stayed up and we also another cell that we call is
style so Winston I might just get you to repeat that if possible we’re losing him
it’s the internet turns out so it’s funny there right as he was talking
about staying online what we found Oh Winston are you back stay back
thank you yes I think you’re back am I better you’re back I used to I
think you’re back like 30 seconds or so okay so we have to optimize so much out
of Magento you know we have changed the way that we run indexes through our site
we have changed the way that we take orders into the cells
orders grid we have killed for example catalog price rules we don’t use that
we’ve killed that off we have looked at specific sequel queries and change that
around and we also have this sale in preparation for Black Friday we have a
sale called a secret sale and on this day we sell
very high-end fashion at insanely low prices so we’ll sell Louis Vuitton
leather boots for like a dollar and on this on this day we get so much traffic
and we use that as a some measuring stick of our MIDI we are for Black
Friday so just out of curiosity how do i how do I get on that secret list just my
just oh you you just have to visit or unless all that is here this it a and
just be a customer we all these shipped to Germany I’m sure I could work better
we’ve known to ship to Germany sure I can bring you a pair of boots or just
have to wear them what with did you mention you built your
own load testing framework could you talk a little bit about that okay so we
had a problem generating the right kind of load on our site so for example we’ve
got react and redux that we use through our site we use Ajax calls quite a bit
and we haven’t found a load testing framework that can interpret JavaScript
and kick off Ajax calls as a normal browser would so what we did was we took
we built a Code section acceptance test and I provision for Amazon x1 32x lodges
and those are insanely big machines I think they got like 500 gigs of memory
hundred and twenty-eight CPUs each and I use this as my load generating service
and then we wrote this code section journey and we use a Google Chrome
hitless browser with a chrome driver interface in between and at the moment
we’re using Python concurrency but we need to generate more concurrency
there’s that we’re looking at swapping it out to go and this is how we load
test our site and generate all that load nice sounds legit
and and baton basically the reason why I said other tools don’t work the the Ajax
requests off of off of your react front-end or whatever they don’t send
the right headers and stuff like that so that the Ajax requests go through well
so a browser will interpret JavaScript and if you open up our homepage for
example we could have for Ajax calls that get kicked off through that into a
profession through JavaScript executing itself right and if you using stuff like
like a Gatling or jmeter I don’t know of a way to kick off those
Java scripts automatically so what it forces us to do is load the home page
then kick off these four Ajax calls then add a product to court then kick off
another Ajax call and you know it’s just a very difficult way of working and it
doesn’t feel like a natural user browsing so with with chrome headless we
plug in a development analytics key into our site and I can see traffic as I
would see normal customer visitation I would see right in our thousand
concurrence in Google Analytics surfing pages as a normal customer would we’ve
added the ability to to choose a random event with random products at between
one two three or four products to the court once those products in the court
go to the to the checkout if they any problems with those products remove
them go add more and then complete this check out check out journey sets it’s
quite interesting yeah that’s awesome so Winston I might just ask you a
question if that’s if that’s possible the hundreds of 50 milliseconds from
Peter Ireland made me think are you looking at progressive web apps at all
like sticking a serviceworker between you and the user yes that is a very good
question we’ve got a rudimentary serviceworker in our site we we looked
at PW is very early on when they’re just attracted mainstream the one problem
that I had was that a PW I when we looked at it would surf style cache and
then as in serviceworker fetch new fresh cache and we have this philosophy where
we need to show absolute freshness on our site at all times and so that cache
caching strategy didn’t fit in with it so what we did was our site is fully
responsive as default two-night of apps iOS and Android and it’s basically just
a native shell with three tabs at the bottom and we embed this sort of ours as
a webview within that within that application that’s given us the ability
to send push notifications you know and Android has become so clever that you
could for example our Contact Us page is a tab click on our number it will bring
up your dollar you keep cool Ricola call center we we’ve got that in
place but we are looking at PW eyes and making this experience with mirror and a
bit seamless and all that kind of stuff yeah nice we did get a clarification
from Oscar on that question about multistage servers he said by multistage
he meant multiple web application server I available service that sounds like you
Winston okay um so multiple web application servers yes so basically you
know we are very conscious of cloud cost and so at different stages of the day
we’ll have different number of servers certainly web front-ends in our cluster
but we typically in the morning 7 o’clock will have one web server at 7
o’clock will automatically will scale up to you to to about 8 that servers that
scaling operation will take about 1 minutes 1 minute and 30 seconds that is
including bringing up the AMI deploying our application on it with 2 health
checks and so roughly about a minute and 30 seconds after that scaling operation
got kicked off you’d start to see traffic in the access log and then what
we do is that is sufficient capacity to ride our node and we thinned as that
load starts to come down a little bit another automated call would get kicked
off and we’d use the scaling policy of AWS to scale us down again to about
three servers and then at 10 o’clock at night we we go in for the night shift
and we kill off two again and we’ll have another server running right through the
night so we call that scheduled scaling but all the while we’ve got dynamic auto
scaling in place so if the CPU of that cluster starts to rise we’ll just add
two more hosts due to that cluster and they will get scaled down dramatically
as this as the load scales up and down so do you mind if I ask is that between
my two and 10 servers or is there sort of an infinite range that you’ll scale
up to I’ve scaled up to a hundred servers before and having a hundred
having a hundred service in your cluster was not something that I would recommend
because it you know we even did deploys and having hundreds and we had to deploy
to ten servers at a time because our because our current deployment framework
uses SSH and it were just the latency we just grow that kind of stuff but our
limit in Amazon is a hundred and fifty but typically you know our approach was
let’s rather have lots of small servers then just a few big ones and I still
like that that approach to this day so we’ll have like a couple of small tiny
teach you leave its T two larges T 2 T two largest in our cluster and you know
we’ll just have a couple of them and we’ll just swap them out as it is and
also another thing that we did like our termination policy is terminate the
oldest first so typically the maximum amount of uptime that a server can have
in my cluster and a front-end server at least is about 12 hours and then that
whole service to take on and her we get a new we get new servers in his place and the other question I would have the
just from a hosting point of view is how do you deal with media the obligatory
horriston question yes sir media was quite an interesting one
I mean media is always a pain and what we wanted to do was we were looking at
EFS the problem that we have is we’ve got this media shared throughout
multiple AWS accounts development staging production and I believe that if
is wasn’t able to do that at that stage it might be able to do it now but what
we did was we install the tiny in if it’s server with the media on a EBS
block device with snapshots daily mounted onto this interface and a share
is exported into different accounts which would then
mount the media onto each of these web hosts as they come up and it’s worked
out it worked out pretty well and what we did was we stuck CloudFront in front
of it as an image CDN so typically in each image would only get fetched once
then it’s in the CDN and from there on it’s just CDN traffic nice what um like
in terms like work-life balance like you know you are with a pretty mature
organization that’s you know you’ve put a TBS through its paces you talked a
little bit about being up at 4:00 a.m. you know some some nights and things
like that what is your work they look like oh you know overall and and and
like how is the work-life balance overall was such massive spikes in
traffic okay so so I’m a I’m a father I’ve got two little children
I’ve got Valerie my daughter she’s she’s four years old and my son Luke he is six
and I’ve got my wife and so my philosophy is that I don’t want to ask
my developers to work outside of work hours they’re allowed to if they’ve been
inspired if they want to continuously ask them to work outside of office hours
something is wrong with the way that I plan my features that I plan my whole
business so by simply there is that portion I believe that you should put
family first you must have a balance and so basically my dad loves to like I wake
up at 7 in the morning I make sure that my cluster is scaled I will convince
opening up at 9 a.m. I take the motorbike to town and I work there until
5:00 5:30 in the evenings and then I’ll come coming to my house and at 6 o’clock
every night we sit down as a family together and we eat and all that kind of
stuff and then and then all that but as a
company we have yoga sessions for both the girls and the guys separately we’re
putting in a gem in our office I’ve got a skipping-rope in my bag I’m learning
artist game I and I look like I can basically imagine Barney skipping that’s
me say you’re not quite up to the like Rocky Balboa or levels I’m getting there
I’m getting there I’m getting there but but we take we take with a good care
with this work-life balance and then also as a team you know once a month
attacked by a whole team and we call it foo-foo Fridays and it’s the last Friday
before payday when you’re at your most broke I think a company I take the
company credit card and my entire team and we go to a remote location we’ve
been at the bay hotel in Kansas Bay and then lunch and drinks for that day
it’s on it’s on runway cell the only requirement that I have is that you work
on something that will benefit yourself and in some way benefit around myself
and y’all so we we try to try to keep it fun and we’re all human you know
that sounds awesome you got are you guys doing visas for people from the United
States out of static curiosity from the Netherlands he’s visited me before we’ve
got a we’ve got a really nice vibe you know it’s awesome our our is there as
there Calicut like a Magento community out there they’re meetups and stuff like
that what city are you in by the way I always forget I mean kata-kata I’m in
Quetta there is a Magento group but it’s not as active and I suppose our
landscape is so different so typically all the Magento developers or other at
agencies or working for merchants such as ourselves
and it’s so competitive the landscape the e-commerce landscape in South Africa
is so competitive that we keep our cords close to our chest I mean I share share
a lot of our stuff but I wouldn’t say there’s a big Magento community here
with meetups every every month and that kind of stuff mmm unfortunately right
right right nice man oh I don’t want to ask so GDP
are I may have violated certain policies announced today has that affected you
guys at all one sinner is is is most of your customer base all South African so
it’s not not too much of an issue look it’s it’s certainly it’s certainly a big
deal for us we we take our customer data pretty seriously and we value our
customers more than anything so we are not in the business of spamming we are
not in a business of of distributing your data we see that as as you know
it’s a it’s it’s it’s what we do and so we we have made precautions to keep your
data safe which my precautions to make it as easy and seamless for you to to
have us not contact you anymore but our site is always there and we’ve got quite
a nice community now once people get in I run my cell
they they love it they love the the process that we offer they love the
product that we have and so they stay close to us
yeah right right and your customer base is pretty much all in South Africa or it
is all in South Africa we only ship within South African borders so we don’t
we don’t ship the ship overseas and we’ve actually said we’ve actually
geared our site in such a way that you have to check out with a South African
address and all that kind of stuff gotcha yeah you got you gotta be careful
with the Europeans man you gotta be careful when you’re doing business out
out that way no I’m just kidding Andrew you we were
talking a little bit about some of the stuff you guys have done around gdpr
yeah we’ve been very careful no there’s the gdpr for us was the early
straightforward it introduces some new requirements particularly around
explicit and easy-to-understand consent which was the thing that you saw that
newsletters pop out for if somebody had just kind of acquired you newsletter
they now needed to ask you explicitly whether or not you wanted to be part of
them so we had to do a few things around contact forms and various other bits and
pieces with that additionally the right to delete and we
need to make sure that we have deletion criteria for all customer data like we
keep invoices for whatever the period is 10 years I think
jenika is interesting because you can you can make a request to delete your
data so for example I am customer and I turn around a Winston and I say I don’t
like you anymore please delete me if I’ve actually made an order of him in
the past then he can’t delete me at least in the case of Germany so you need
to make sure that you can build that logic one of the things I was curious
about that so the the way it’s written in the law of the relevant country like
you’re allowed to not allowed to keep data except where other law requires you
to explicitly write so employment contracts I think or the other one
that’s pretty you need to keep those there’s a few pieces so it was a bit of
a non-event for us we just kind of said yeah I think I think I saw something on
Twitter over the weekend that like a bunch of politicians have requested
articles that weren’t flattering about them to be deleted from Google there are
30,000 deletion requests or something like that so that’s okay I didn’t know that such
I’m allowed to do that the other one that’s a super interesting case is to
whiz records if you query who is records now for you there will be nothing and
they’re not allowed to show us data this one yeah
so it’s like old things a double-edged sword I think there are provisions in
the GDP are for the public interest but I’m certainly not up to date on what
they are so yeah crazy yeah it’s gonna be interesting to see how everything
shakes out how do you laws in and obviously once I know you’re not a
lawyer and everything like that but how do how do laws in South Africa compared
to this stuff in the GDP are is there are there was a molarity Zoar
well said Africa’s as started to make headway with that you know we were very
much an evolving market a and so so we for example would get a lot of calls at
any time of the day from anyone that has ever found your details so for example I
don’t even check my SM ASIS anymore because I get like a hundred today maybe
not that much I get I get like 20 30 SMSs a day and it could range anything
from funeral cover to you’ve been accepted for a loan to life cover to you
know get get less insurance – I know a TV licence that’s worse than the u.s.
man I don’t get that much SMS everything so my mother always says to me like you
never read my essay basis well sorry okay you we have to talk we talk you
know and they know and we I get to be on the story because as SMS has just get
lost so for the most part I a check what’s apps and all that kind of stuff
but for the most part the law is heading in the right direction but it’s
certainly very loose at the moment right it’s like the wall the wall twists
right sounds like we’re cool man any anything else we haven’t touched on yet
that uh yeah that we want to get to before we wrap this bad boy up
obligatory we’re hiring please come and see you I suppose awesome
Winston anything else kind and nothing from us odd man I just want to say
Andrew it’s been you’re you’re an encyclopedia of talent I always enjoy
our conversations you know a great deal about hosting kubernetes all that kind
of stuff kind and thank you for having me on your man it’s it’s been a real
pleasure thank you yeah absolutely thanks has a lot of fun and thanks
everybody for tuning in if you have any other questions that weren’t answered
you can shoot me an email or something like that and I’ll I’ll chase them down
get some answers for them where can people find you guys online on the
Internet Winston you have sort of a Twitter
account ish but it’s I don’t know if you use it other than just posting your
music and stuff like that where we just I just I can be reached at developer
beats or where you can just pretty much find the invites chat I guess that’s a
good place to start but I don’t really tweet about tech I’ll just tweet good
sits that and I enjoy listening to while I’m writing code and all that kind of
stuff nice and Andrew what’s your what’s your Twitter handle these days and
you’re helping comma because because my name is Andrew Hadden and you can go to
calm ah tricky tricky tricky very nice awesome well thanks guys
things everybody for tuning in we’ll see you very soon and that’s it