Cloudera

JULY 13, 2021

After the launch of CDP Data Engineering (CDE) on AWS a few months ago, we are thrilled to announce that CDE, the only cloud-native service purpose built for enterprise data engineers, is now available on Microsoft Azure. . Prerequisites for deploying CDP Data Engineering on Azure can be found here.

Data Engineering 127

Data Engineering Azure Engineering Enterprise 127

Netflix Tech

DECEMBER 14, 2023

Engineers from across the company came together to share best practices on everything from Data Processing Patterns to Building Reliable Data Pipelines. The result was a series of talks which we are now sharing with the rest of the Data Engineering community! In this video, Sr.

Data Engineering 106

Data Engineering Engineering Data Software Engineering 106

Altexsoft

JUNE 25, 2019

If we look at the hierarchy of needs in data science implementations, we’ll see that the next step after gathering your data for analysis is data engineering. This discipline is not to be underestimated, as it enables effective data storing and reliable data flow while taking charge of the infrastructure.

Data Engineering 106

Data Engineering Engineering Data Artificial Inteligence 106

Xebia

MARCH 2, 2025

Deployment isolation: Handling multiple users and environments During the development of a new data pipeline, it is common to make tests to check if all dependencies are working correctly. However, we want to test our workflow logic faster during development, and waiting times are frustrating. x-cpu-ml-scala2.12

Resources 130

Resources Testing Metrics Data Engineering 130

CIO

MARCH 13, 2025

And right now, theres no greater test of that than AI. Mike Vaughan serves as Chief Data Officer for Brown & Brown Insurance. Innovate and explore Use technology to drive better outcomes and future-proof our business. The real challenge is balancing all three while ensuring innovation leads the way.

Company 186

Company Generative AI Insurance Education 186

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. Test Drive CDP Pubic Cloud.

Data Engineering 118

Data Engineering Technical Review Software Review Engineering 118

Xebia

OCTOBER 18, 2024

Dbt is a popular tool for transforming data in a data warehouse or data lake. It enables data engineers and analysts to write modular SQL transformations, with built-in support for data testing and documentation. This makes dbt a natural choice for the Ducklake setup.

Open Source 162

Open Source AWS Government Technical Review 162

Xebia

NOVEMBER 21, 2024

hooks: - id: check-model-has-tests args: ["--test-cnt", "2", "--"] While dbt-checkpoint offers numerous useful hooks, it is limited by the fact that it is designed to work as a pre-commit hook. Tests can be added for models, documentation coverage and best practices like avoiding chained views.

Weak Development Team 130

Weak Development Team Testing Analytics Engineering 130

Altexsoft

OCTOBER 30, 2021

Engineers are not only the ones bearing helmets and operating on construction sites. Scientists don’t always wear lab coats or handle test tubes. Explaining the difference, especially when they both work with something intangible such as data , is difficult. Data science vs data engineering.

Data Engineering 98

Data Engineering Engineering Data Machine Learning 98

CIO

NOVEMBER 7, 2024

It covers essential topics like artificial intelligence, our use of data models, our approach to technical debt, and the modernization of legacy systems. We explore the essence of data and the intricacies of data engineering. We are also testing it with engineering.

Artificial Inteligence 180

Artificial Inteligence Development Generative AI Artificial Intelligence 180

Cloudera

SEPTEMBER 2, 2021

At Cloudera, we introduced Cloudera Data Engineering (CDE) as part of our Enterprise Data Cloud product — Cloudera Data Platform (CDP) — to meet these challenges. We tested the scaling capabilities of CDE with the following job runs to mimic a real-world scenario: . fixed sized clusters). What’s next.

Data Engineering 85

Data Engineering Performance Engineering Data 85

Netflix Tech

JULY 15, 2021

Data Engineers of Netflix?—?Interview Interview with Kevin Wylie This post is part of our “Data Engineers of Netflix” series, where our very own data engineers talk about their journeys to Data Engineering @ Netflix. Kevin, what drew you to data engineering?

Data Engineering 79

Data Engineering Engineering Data Big Data 79

Xebia

DECEMBER 26, 2024

Data workers can deploy their resources to a development workspace to test their application. After testing, you can integrate your bundle to a CI/CD pipeline to make deployment to a production environment. You are ready to run and test your application logic. Resources are defined in a readable format (YAML files).

Resources 130

Resources Testing Infrastructure Applications 130

AWS Machine Learning - AI

MARCH 13, 2025

This post was co-written with Vishal Singh, Data Engineering Leader at Data & Analytics team of GoDaddy Generative AI solutions have the potential to transform businesses by boosting productivity and improving customer experiences, and using large language models (LLMs) in these solutions has become increasingly popular.

Artificial Inteligence 96

Artificial Inteligence Systems Review System Generative AI 96

The Crazy Programmer

NOVEMBER 6, 2021

They are responsible for designing, testing, and managing the software products of the systems. Big Data Engineer. Another highest-paying job skill in the IT sector is big data engineering. And as a big data engineer, you need to work around the big data sets of the applications.

Artificial Inteligence 290

Artificial Inteligence Blockchain Software Review Artificial Intelligence 290

Cloudera

OCTOBER 19, 2021

Also there’s opportunities for us to provide read-to-use pipeline definitions that capture very common patterns such as detecting files on S3 bucket, running data transformation with Spark, and performing data mart creation with Hive. . When creating a Virtual Cluster a new option will allow the enablement of the Airflow authoring UI.

Data Engineering 120

Data Engineering Engineering Data Virtualization 120

CIO

OCTOBER 13, 2023

Big data architect: The big data architect designs and implements data architectures supporting the storage, processing, and analysis of large volumes of data. Data architect vs. data engineer The data architect and data engineer roles are closely related.

Data 199

Data Data Engineering Database Administration Artificial Inteligence 199

CIO

FEBRUARY 19, 2025

EYs Gusher says shes seeing gen AI value in code debugging and testing. Weve also seen some significant benefits in leveraging it for productivity in data engineering processes, such as generating data pipelines in a more efficient way.

Google Cloud 206

Google Cloud Survey CTO Coach Software Development 206

Xebia

JULY 19, 2023

As we depend more on these systems, testing should be a top priority during deployment. AI systems are even more vulnerable as, besides code, they leverage data and algorithms, so you need to test all the components to avoid whammies. When a new system version is ready, the tests ensure it still functions correctly.

Artificial Inteligence 130

Artificial Inteligence Systems Review System Testing 130

Xebia

JANUARY 9, 2024

Yes, dbt does provide logs to the stdout for every model and test execution, however in my opinion this is not sufficient to base your whole monitoring around. These log records will show the name of the model or test, the execution time, and the execution status (passed, warned, or failed). Whenever dbt runs (e.g.

Testing 130

Testing Data Open Source Applications 130

InfoQ Culture Methods

FEBRUARY 22, 2022

Software testing, especially in large scale projects, is a time intensive process. Test suites may be computationally expensive, compete with each other for available hardware, or simply be so large as to cause considerable delay until their results are available.

Artificial Inteligence 109

Artificial Inteligence Machine Learning Testing Development 109

InfoQ Culture Methods

FEBRUARY 24, 2023

In this article, Tariq King describes the metaverse concept, discusses its key engineering challenges and quality concerns, and then walks through recent technological advances in AI and software testing that are helping to mitigate these challenges. By Tariq King

Testing 121

Testing Engineering Software Technology 121

InfoQ Culture Methods

MAY 4, 2020

We will demystify AI, and see how it is already embedded in our everyday life, and then you are going to learn about how we (The folks at Testim.io) utilised this kind of groundbreaking technology to bring test automation to the next level. By Daniel Gold.

Testing 127

Testing Technology Data Engineering Artificial Inteligence 127

Perficient

FEBRUARY 13, 2023

Introduction: We often end up creating a problem while working on data. So, here are few best practices for data engineering using snowflake: 1.Transform This makes it easier to test intermediate results, simplifies code, and often produces simpler SQL code that runs faster.

Data Engineering 69

Data Engineering Engineering Data Storage 69

CIO

JUNE 7, 2022

You don’t understand how long you should test your feature and what exactly you should measure,” he says. ML engineer. Data scientists may build the ML models, but its ML engineers who implement them. “An An ML engineer is also involved with validation of models, A/B testing, and monitoring in production.”.

Artificial Inteligence 218

Artificial Inteligence Technical Review Fractional CTO Data Engineering 218

CIO

FEBRUARY 25, 2025

Being data-forward isnt just about technology. Its about being willing to test hypotheses, learn from the results and continuously improve. Mike Vaughan serves as Chief Data Officer for Brown & Brown Insurance. Its about aligning people, processes and purpose to drive meaningful outcomes.

Data 136

Data Innovation Insurance Culture 136

CIO

NOVEMBER 27, 2024

Not cleaning your data enough causes obvious problems, but context is key. But making data too uniform can lead to models that perform well on clean, structured data like their training set, but struggle with real-world messy data, giving you poor performance in production environments.

Data 211

Data Enterprise Weak Development Team Software Review 211

Gorilla Logic

APRIL 27, 2022

Are you a data engineer or seeking to become one? This is the first entry of a series of articles about skills you’ll need in your everyday life as a data engineer. This blog post is for you. So let’s begin with the first and, in my opinion, the most useful tool in your technical tool belt, SQL. CROSS JOIN.

Data Engineering 59

Data Engineering Engineering Data Windows 59

CircleCI

JULY 29, 2022

The data world has adopted software development practices in recent years to test data changes before deployment. The testing process can be time-consuming and prone to unexpected errors. For example, at CircleCI, our data team uses dbt at scale. Why is dbt useful in data engineering and analysis?

Testing 98

Testing Software Review Continuous Integration Software Development 98

Mobilunity

MAY 3, 2024

When it comes to financial technology, data engineers are the most important architects. As fintech continues to change the way standard financial services are done, the data engineer’s job becomes more and more important in shaping the future of the industry.

Data Engineering 52

Data Engineering Fintech Engineering Data 52

TechCrunch

MARCH 21, 2023

Organizations dealing with large amounts of data often struggle to ensure that data remains high-quality. According to a survey from Great Expectations, which creates open source tools for data testing, 77% of companies have data quality issues and 91% believe that it’s impacting their performance.

Data 202

Data Data Engineering Training Engineering 202

CIO

JANUARY 6, 2023

Database developers should have experience with NoSQL databases, Oracle Database, big data infrastructure, and big data engines such as Hadoop. The role typically requires a bachelor’s degree in computer science, electrical engineering, computer engineering or a related discipline.

LAN 215

LAN How To Systems Administration Software Engineering 215

CIO

JUNE 10, 2024

For example, Napoli needs conventional data wrangling, data engineering, and data governance skills, as well as IT pros versed in newer tools and techniques such as vector databases, large language models (LLMs), and prompt engineering. Meanwhile, 54% of respondents said skills shortages hamper change.

Artificial Inteligence 204

Artificial Inteligence Strategy Machine Learning Training 204

Xebia

DECEMBER 22, 2023

Some examples: It’s not uncommon for us to observe a ‘testing’ status to take longer to complete than the actual implementation, often this relates to hand-offs, poor testability, or an inefficient test strategy. Refinement status might be overly short or skipped over entirely. Is our backlog management efficient enough?

Weak Development Team 244

Weak Development Team Data Trends Software Review 244

TechCrunch

OCTOBER 5, 2021

The team noted at the time that the current process for interviewing software engineers didn’t really work for measuring how well someone would do in a day-to-day engineering job. Image Credits: Byteboard.

Technical Review 233

Technical Review Software Review PHP Software Engineering 233

CIO

OCTOBER 20, 2023

The vendor-neutral certification covers topics such as organizational structure, security and risk management, asset security, security operations, identity and access management (IAM), security assessment and testing, and security architecture and engineering.

SCRUM 206

SCRUM Azure AWS Agile 206

O'Reilly Media - Data

APRIL 3, 2018

Data Science and Machine Learning sessions will cover tools, techniques, and case studies. This year’s sessions on Data Engineering and Architecture showcases streaming and real-time applications, along with the data platforms used at several leading companies. Privacy and security.

Artificial Inteligence 240

Artificial Inteligence Machine Learning Company Case Study 240