Streamdal is an open-source platform for managing code-native data pipelines | Join us 🤝 discord.gg/streamdal 🌟 github.com/streamdal/streamd…

Joined October 2012
77 Photos and videos
Pinned Tweet
25 Jan 2024
PII Protection Upgrade! 🛡️ We’re thrilled to unveil the latest feature for the Streamdal pipeline engine: ✨ Automatic Detection and Masking of PII ✨ Much less manual hassle! With a few clicks, safeguard sensitive info like email addresses or phone numbers automatically in your data pipelines. 🚀 🔐 Streamlined Security: Effortlessly obscure PII data to and from external systems. 🔧 Simplified Workflows: Greatly simplify custom steps; this feature detects PII dynamically in payloads! 🎯 Precision Matters: Tested rigorously to detect and mask PII, even in complex JSON structures. 📹 Curious? @jacobheric got a quick video walkthrough for you! #DataSecurity #PII
2
7
437
Streamdal retweeted
18 Apr 2024
💡 Ecosystem highlight of the week: prisma-extension-streamdal @streamdal is an open-source tool to embed privacy controls into app code. With this Prisma Client extension, you can code native data pipelines for Prisma Client queries. 👉 Check it out at github.com/streamdal/prisma-…
2
4
20
4,086
22 Mar 2024
💡Streamdal is all about automating mundane and annoying tasks… That's why we're excited to introduce our newest enhancement 🎉: a Terraform provider used for stamping down pipelines and various data rules - no more manual clicking around in a web UI! 👀Check out our blog post by our Lead Software Engineer, Mark Gregan (@marqueetag) for a step-by-step guide on getting your pipeline setup with our Terraform provider (the link is in the following thread) 🔗👇 #OpenSource #Terraform #CodeNative #PII #DataPrivacy
1
3
5
358
Streamdal retweeted
21 Mar 2024
🔍 Curious about how Streamdal's mission to democratize real-time data pipelines connects with our latest update? 🌐 Dive into our blog post to uncover the magic! ✨ 🌟 Discover how our open-source shim for KafkaJS is revolutionizing data pipelines, empowering any application to seamlessly integrate sophisticated functionalities with just a few configuration tweaks 🚀💻 💡 Ready to streamline your data operations and unlock new possibilities? Check the links in the following thread 🔗 👇 #OpenSource #Kafka #JavaSkript #DataEngineering #DataQuality
1
3
6
196
21 Mar 2024
🔍 Curious about how Streamdal's mission to democratize real-time data pipelines connects with our latest update? 🌐 Dive into our blog post to uncover the magic! ✨ 🌟 Discover how our open-source shim for KafkaJS is revolutionizing data pipelines, empowering any application to seamlessly integrate sophisticated functionalities with just a few configuration tweaks 🚀💻 💡 Ready to streamline your data operations and unlock new possibilities? Check the links in the following thread 🔗 👇 #OpenSource #Kafka #JavaSkript #DataEngineering #DataQuality
1
3
6
196
21 Mar 2024
Your favorite libraries, but now with Streamdal! We are on a roll with these, and plan to continue to pump these out regularly: 🌟streamdal/ibm-sarama 🌟streamdal/kafkajs 🌟streamdal/segmentio-kafka-go 🌟streamdal/confluent-kafka-python 🌟streamdal/confluent-kafka-go
1
42
14 Mar 2024
🔮 Imagine a dev world where data privacy and compliance changes are made immediately and directly in your app, development is empowered, and costs are slashed. 👀Check out the future of data privacy - our CTO Dan Selans (@skey) breaks down the code-native concept Find the link in the thread to get an idea about this modern approach to dealing with data privacy-related issues 🔗 👇 #DataPrivacy #SoftwareDevelopment #SoftwareEngineering
1
2
4
85
13 Mar 2024
🔐Want to ensure data is protected in dev? Our Director of Engineering, @fritzstauff shows you how to get powerful data workflows up and running quickly in your development environments 🚀 Check it out 👇 #Development #TechTutorial #DataPrivacy #PII #CodeNative
2
3
74
12 Mar 2024
🛡️Tired of playing catch-up with data security drama? Streamdal CTO, Dan Selans (@skey), lays down the facts on Upstream Data Security. Learn what it’s all about and why you should care about it. Dive into the insights 🔗👇 #DataPrivacy #PII #DataEngineering #Governance
2
1
5
95
6 Mar 2024
🚨 Video Walkthrough Alert Join our lead engineer, @jacobheric, as he takes you through the newest feature on the Streamdal Code-Native platform: HTTP Requests. Explore this exciting addition 👇 #CodeNative
2
7
70
29 Feb 2024
Ready for full Node mode? Whether you need to observe, operate on data, or are looking for something clean and fresh for your project, check out our Node SDK. Similar to our other SDKs, it’s open-source and used to enable data observability and code-native data pipelines. You’ll find this tech in it: ✅Protobuf - Typescript/Javascript gRPC and protocol buffers for Node. ✅Vitest (@vitest_dev) & Sinon - Great for testing! And of course, a lot more 🔗👇 #FOSS #FossFeb #opensource #Node #protobuf
1
2
3
117
29 Feb 2024
Check out the Node SDK on Github: github.com/streamdal/node-sd… Other useful links 👇 Protobuf: github.com/protocolbuffers/p… Sinon: sinonjs.org/ Vitest: vitest.dev/
1
1
34
28 Feb 2024
Thrilled to share a recent endeavor – a comprehensive video walkthrough showcasing the ease of extracting, truncating, and adding metadata with Streamdal 🎥✨ This powerful technique allows you to easily filter out messages or data that may not be intended for downstream endpoints such as Logstash, Snowflake, S3, Postgres, and more. Check it out! 🚀 #DataEngineering #DevOps #DataManagement #Engineering
1
4
7
758
26 Feb 2024
More @golang inspiration for those developers young and old, learned or aspiring! To visualize read/write, observe data, and sync rules for workflows and pipelines, we needed a way to communicate with the frontend. We built that server functionality in Go. Like all of our components, the server is open-source. Some particulars you’ll see: ✅gRPC (@grpcio) Protobuf - We use both unary and streaming functionality for communication. ✅Redis (@Redisinc) - We specifically use redis for Pubsub and its traditional key/value functionality. ✅Go-director - This library enables us to declare for loops as injectable dependencies, which greatly simplifies testing. The code 🔗👇 #FOSS #FossFeb #opensource #go #redis
2
1
5
77
23 Feb 2024
🤔 Data anonymization vs redaction when handling PII - Is one more ideal than the other? They both play an important role, but it's all about context! 🔴 Redact PII for sharing & compliance. 🟢 Anonymize PII for long-term storage & utility. You should redact PII or PHI fields in data when: ✅Sending data to 3rd parties ✅Complying with regulations like HIPAA ✅Ensuring readability while preserving data context You should anonymize or obfuscate PII or PHI fields in data when: ✅Storing for long-term retention ✅Complying with regulations like GDPR ✅Needing to preserve data utility Your choice hinges on data and regulations. Tools like Streamdal give you the best of both worlds. #DataProtection #PII
3
7
320