Last week I spent some time to understand the #PySpark applyInPandasWithState. This week I'm refactoring the code, hoping to still understand it 2 months later ;) 👉 waitingforcode.com/apache-spark-s…
0
0
2
In the previous release #PySpark has got an interesting streaming feature -> the arbitrary stateful processing. It has a different API than the Scala version but is more adapted to the Python world.
More 👉 waitingforcode.com/apache-spark-s…
More 👉 waitingforcode.com/apache-spark-s…
0
0
1
It's not a rebranding but more a regrouping 😉 All my additional #dataengineering content is now available from there waitingforcode.com/better (planning to add some stream processing materials soon)
0
0
5
A new module of the Become a Data Engineer V2 is online. You can now enroll in and learn stream processing. Advanced stateful processing concepts are will be released next 👉 becomedataengineer.com/#module-4
0
1
3
Next module of the reworked Become a Data Engineer V2 is online. You can now enroll in and learn about data cleansing. The streaming modules are next to be released, soon!
👉 becomedataengineer.com/#details
👉 becomedataengineer.com/#details
0
0
6
Arbitrary stateful processing is no more Scala-reserved! It's one of the 3 new features available in #ApacheSpark Structured Streaming 3.4.0 and presented in the new blog post 👉 waitingforcode.com/apache-spark-s…
0
1
7
Writing this #ApacheSpark 3.4.0 blog post has taken me the most time so far 🥵 But it helped understand the global watermark issue and the fixes reducing its impact added in the recent release 👉 waitingforcode.com/apache-spark-s…
1
0
5
I'm slowly but regularly writing updates on the #ApacheSpark 3.4.0. To start, there will be 3 blog posts about new features in Structured Streaming. The first of them about async progress tracking is already there 👉 waitingforcode.com/apache-spark-s… Enjoy!
0
1
7
Amazon Kinesis confidences, part 2 aka how I learned the difference between #ApacheKafka offset and Amazon Kinesis SequenceNumber 👉 waitingforcode.com/data-engineeri…
0
0
2
Despite some high-level similarities, #ApacheKafka and #AWS Kinesis Data Streams are different! Despite both being streaming brokers, they have different scaling policies, supported delivery semantics, ... and many others available in the new blog post 👉 waitingforcode.com/data-engineeri…
0
1
6
As promised, I'm continuing with a new series called "Worth reading for data engineers". It summarizes interesting blog posts I've read. This time you'll find a lot of streaming and a bit of project organization 👉 waitingforcode.com/general-data-e…
0
1
11
Athena for GCS data? An autoscaling slot reservation for BigQuery? These are only 2 changes on the cloud for data engineers. You'll find the rest in the next part of the blog post series "What's new on the cloud for data engineers..." 👉 waitingforcode.com/data-engineeri…
0
0
2
Filtering irrelevant records out is a common step in data pipelines. But how to know what is the filter rule filtering out the most of the input data 🤔One approach is to use accumulators in #ApacheSpark. An example in the new blog post 👉 waitingforcode.com/apache-spark-s…
1
1
2
Next week I will resume blogging. This time, the goal is to publish at least 1 blog post per week.
Meantime, if you are looking for some reading, the "Data engineering patterns on the cloud" ebook is ready and available on Leanpub 👉 leanpub.com/data-engineeri…
See you next week!
Meantime, if you are looking for some reading, the "Data engineering patterns on the cloud" ebook is ready and available on Leanpub 👉 leanpub.com/data-engineeri…
See you next week!
0
2
15
The last catch-up tweet. I'm in San Francisco to attend the #DataAISummit in-person! After solving the check-in problems I could take a walk and appreciate the landscape yesterday. Thank you @dennylee @jaceklaskowski @laduramvishnoi for your help 🙏 See you at the conference!
0
0
4
This year I've plans to do some side-projects besides blogging. The first one has already started! To get a better idea of it, you can download the first chapter of "Data engineering patterns on the cloud" here 👉 dataengineeringpatternsonthecloud.com
0
5
35
Since last week I've had only 1 hand left for blogging. I'll take the month off to learn blogging in this new wonderful context! The new blog post should be there in October then🤞
11
0
36
🌞 The Summer edition of Become a #DataEngineer course is open! If you want to join, use this special link becomedataengineer.com/?coupon_code=B… by the 7th of July.
0
1
11
Even though I'm happy to share new blog posts, this time I've something else 😀 Today, one week after passing DP-200, I passed DP-201 and became a certified #Azure Data Engineer! Soon I will share my preparation process and start to work on step 2!👉 credly.com/badges/f270e9a…
0
1
10
The Spring edition of Become a Data Engineer is now open! You can enroll in the course during the next 11 days with the help of this special link becomedataengineer.com/?coupon_code=B… 🎁
0
1
4
☃️ Winter edition of Become a #DataEngineer course starts today. If you want to join, you can use this special link becomedataengineer.com/?coupon_code=B… before February 11.
0
1
6
I got the confirmation this morning and can officially share the news! I successfully passed the #GCP Data Engineer certification 😀 The blog post about my learning process will be online next month. Feel free to ask any question, will try to answer in January's blog post.
4
0
23
Autumn edition of Become a #DataEngineer course starts today and ends next Sunday. If you want to join, you can use this special link becomedataengineer.com/?coupon_code=B… 🎁
1
0
5
Had a nice surprise this morning ☕ Happy to see the blog helped and many thanks Christopher for your support 👍
0
0
1
One week left before the end of the Summer Edition of Become a Data Engineer ⌛ You will find all necessary information and a gift on this page 👉 becomedataengineer.com/?coupon_code=B… [PS. Do not cut the link, otherwise the gift will disappear 👻 ]
0
2
10
Transparent png should be forbidden 😝 I couldn't resist to play a little after making the last checks of becomedataengineer.com video thumbnails before the next week's opening. Let me introduce a Barkula, Bartrooper and piBarte. Serious-me will be back tomorrow ;)
0
0
2
Winter is coming 🥶 And it's a good time to make new 📚 friends:
@therealdatabass by @ifesdjeen
"Foundations for Architecting Data Solutions" by @jseidman and @TedMalaska
"Enterprise Integration Patterns" by @ghohpe
"The Software Craftsman" by @sandromancuso
@therealdatabass by @ifesdjeen
"Foundations for Architecting Data Solutions" by @jseidman and @TedMalaska
"Enterprise Integration Patterns" by @ghohpe
"The Software Craftsman" by @sandromancuso
1
0
12
Playing with Selector to better understand Kafka in-flight requests. I tried to write a simple client-server app but it didn't work. The client was always registered but had 0 keys after .select() call.
That's normal when you use server selector in the client 🤦♂️ #copyPasteFail
That's normal when you use server selector in the client 🤦♂️ #copyPasteFail
0
0
1
Leaving #Amsterdam after 2 great days with #ApacheSpark. And yesterday after my #SparkAISummit talk I definitevely left my comfort zone! If you want to leave yours, take a not reachable goal today and reach it tomorrow. OK, I'm going too philosophical so I will stop now :P
1
0
5
Safely arrived in #Amsterdam for #SparkAISummit. Never seen as many bikes like during last 20 minutes 🚲 🚲🚲 🚲
See you tomorrow at the Summit!
See you tomorrow at the Summit!
1
0
8
Yesterday I was playing with #intelliJIDEA and by chance I discovered a feature to log whatever happens inside the debugged code 🤩 Very useful to confirm different hypotheses about how some code is working. Search for "log and evaluate".
0
2
10
Reading plan for April = @ManningBooks + @streamingbook . Soon on waitingforcode.com I will publish the reviews of books I read - mostly data, cloud & software engineering . Stay tuned ! 📚
1
0
6
After several months passed in my backlog, #graph s are coming as a new #data category in waitingforcode. There is nothing better than a post with some generic information to start the learning process calmly,: waitingforcode.com/graphs/graphs-…
0
0
0
This week I wanted to explore #Scala feature to generate and execute the code dynamically. Quasiquotes are an easy way to do that. All observations are noted in just published post from #OneScalaFeaturePerWeek series waitingforcode.com/scala-core/qua…
0
0
0
The technology behind #Redshift and Azure Data Warehouse explained in today's post about MPP waitingforcode.com/general-big-da…
0
0
0
The last of recently covered Bloom filters - Stable Bloom filter. It's an another approach to apply the filter to unbounded data. More explanation in today's post waitingforcode.com/big-data-algor…
0
0
2
Today in #OneScalaFeaturePerWeek some nodes about the complexity of #Scala collections waitingforcode.com/scala-collecti…
0
0
0
Explanation of another #BloomFilter type - Scalable Bloom filter in one of today's post waitingforcode.com/big-data-algor…
0
0
1
The first post from series about Bloom filters explaining their most basic version waitingforcode.com/big-data-algor…
0
1
2
Another post explaining #Gnocchi internals. This time it's about reading aggregates waitingforcode.com/time-series/re…
0
0
1
In another today's post some notes about #ApacheSpark external shuffle service: waitingforcode.com/apache-spark/e…
0
1
2
Correlated scalar subqueries are also present in #ApacheSparkSQL. Some examples in today's post waitingforcode.com/apache-spark-s…
0
0
0
Another part of #Gnocchi learning. This time a focus on Carbonara, Gnocchi's file format for time series waitingforcode.com/time-series/ca…
0
0
0
After talking about #timeSeries databases, it's time to introduce #Gnocchi architecture waitingforcode.com/time-series/gn…
0
0
0
New part about the Conflct-Free Replicated Data Types (#CRDT) - flags, graphs and maps waitingforcode.com/big-data-algor…
0
0
3
Another post about a #BigData approximation algorithm on waitingforcode. This time the Count-min sketch explained here waitingforcode.com/big-data-algor…
0
0
0
Just discovered by mistake a behavior of UNION's in SQL when the aliased fields are not ordered in the same way in both SELECTs:
mysql> SELECT "a1" AS a, "b1" AS b UNION ALL SELECT "b2" AS b, "a2" AS a;
The output won't be a: a1, a2, b: b1, b2, but
mysql> SELECT "a1" AS a, "b1" AS b UNION ALL SELECT "b2" AS b, "a2" AS a;
The output won't be a: a1, a2, b: b1, b2, but
0
0
0
The 1st post about #Scala and its immutable collections complexity waitingforcode.com/scala-collecti…
0
0
1
