Xebia

JANUARY 19, 2023

The following is a review of the book Fundamentals of Data Engineering by Joe Reis and Matt Housley, published by O’Reilly in June of 2022, and some takeaway lessons. This book is as good for a project manager or any other non-technical role as it is for a computer science student or a data engineer.

Data Engineering 130

Data Engineering Engineering Data Technical Review 130

Erik Bernhardsson

DECEMBER 15, 2020

It's a popular attitude among developers to rant about our tools and how broken things are. I had my first job as a software engineer in 1999, and in the last two decades I've seen software engineering changing in ways that have made us orders of magnitude more productive. The insatiable demand for software.

Software Engineering 325

Software Engineering Engineering Tools Software 325

CIO

JANUARY 3, 2025

Today, IT encompasses site reliability engineering (SRE), platform engineering, DevOps, and automation teams, and the need to manage services across multi-cloud and hybrid-cloud environments in addition to legacy systems. At the same time, the scale of observability data generated from multiple tools exceeds human capacity to manage.

Organization 158

Organization Artificial Intelligence Artificial Inteligence DevOps 158

Xebia

APRIL 9, 2025

For example, events such as Twitters rebranding to X, and PySparks rise in the data engineering realm over Spark have all contributed to this decline. In my opinion, sbt (Simple Build Tool) is a perfect example of this evolution. Various business decisions have altered its public perception.

Systems Review 130

Systems Review Programming Technical Review Engineering 130

CIO

DECEMBER 10, 2024

to GPT-o1, the list keeps growing, along with a legion of new tools and platforms used for developing and customizing these models for specific use cases. To integrate AI into enterprise workflows, we must first do the foundation work to get our clients data estate optimized, structured, and migrated to the cloud. From Llama3.1

Artificial Inteligence 174

Artificial Inteligence Off-The-Shelf Insurance Analytics 174

CIO

APRIL 1, 2025

Job titles like data engineer, machine learning engineer, and AI product manager have supplanted traditional software developers near the top of the heap as companies rush to adopt AI and cybersecurity professionals remain in high demand. Theres real hand-holding that needs to be done.

Marketing 152

Marketing Software Development Software Development 152

TechCrunch

DECEMBER 1, 2020

DevOps continues to get a lot of attention as a wave of companies develop more sophisticated tools to help developers manage increasingly complex architectures and workloads. “Users didn’t know how to organize their tools and systems to produce reliable data products.”

Tools 214

Tools Data Weak Development Team Big Data 214

Altexsoft

JUNE 25, 2019

If we look at the hierarchy of needs in data science implementations, we’ll see that the next step after gathering your data for analysis is data engineering. This discipline is not to be underestimated, as it enables effective data storing and reliable data flow while taking charge of the infrastructure.

Data Engineering 106

Data Engineering Engineering Data Artificial Inteligence 106

Cloudera

DECEMBER 21, 2021

Since the release of Cloudera Data Engineering (CDE) more than a year ago , our number one goal was operationalizing Spark pipelines at scale with first class tooling designed to streamline automation and observability. Data pipelines are composed of multiple steps with dependencies and triggers. New in 2021.

Data Engineering 118

Data Engineering Technical Review Software Review Engineering 118

CIO

NOVEMBER 7, 2024

It’s federated, so they sit in the different business units and come together as a data community to harness our full enterprise capabilities. On the technology side, we think about the engineering aspects — access, platforms, tools, insights, transformations, all these different components to it. So that’s the journey we’re on.

Artificial Inteligence 180

Artificial Inteligence Development Generative AI Artificial Intelligence 180

O'Reilly Media - Ideas

JULY 3, 2019

The O'Reilly Data Show: Ben Lorica chats with Jeff Meyerson of Software Engineering Daily about data engineering, data architecture and infrastructure, and machine learning. Their conversation mainly centered around data engineering, data architecture and infrastructure, and machine learning (ML).

Machine Learning 22

Machine Learning Artificial Inteligence Tools Development 22

CIO

OCTOBER 13, 2023

The data architect also “provides a standard common business vocabulary, expresses strategic requirements, outlines high-level integrated designs to meet those requirements, and aligns with enterprise strategy and related business architecture,” according to DAMA International’s Data Management Body of Knowledge.

Data 199

Data Data Engineering Database Administration Artificial Inteligence 199

O'Reilly Media - Ideas

DECEMBER 6, 2018

As the use of machine learning and analytics become more widespread, we’re beginning to see tools that enable data scientists and data engineers to scale and tackle many more problems and maintain more systems. Continue reading Tools for generating deep neural networks with efficient network architectures.

Network 86

Network Architecture Tools Artificial Inteligence 86

CIO

JANUARY 22, 2025

CIOs should also build platforms for custom tools that meet the specific needs not only of their industry and geography, but of their company and even for specific divisions. AI models will be developed differently for different industries, and different data will be used to train for the healthcare industry than for logistics, for example.

Weak Development Team 192

Weak Development Team Education Meeting Data 192

Cloudera

FEBRUARY 14, 2022

When we introduced Cloudera Data Engineering (CDE) in the Public Cloud in 2020 it was a culmination of many years of working alongside companies as they deployed Apache Spark based ETL workloads at scale. Each unlocking value in the data engineering workflows enterprises can start taking advantage of. Usage Patterns.

Data Engineering 113

Data Engineering Engineering Data Storage 113

CIO

DECEMBER 22, 2022

DataOps (data operations) is an agile, process-oriented methodology for developing and delivering analytics. It brings together DevOps teams with data engineers and data scientists to provide the tools, processes, and organizational structures to support the data-focused enterprise. What is DataOps?

Analytics 195

Analytics Data Engineering Artificial Inteligence Machine Learning 195

Altexsoft

OCTOBER 30, 2021

If you’re an executive who has a hard time understanding the underlying processes of data science and get confused with terminology, keep reading. We will try to answer your questions and explain how two critical data jobs are different and where they overlap. Data science vs data engineering.

Data Engineering 98

Data Engineering Engineering Data Machine Learning 98

Cloudera

SEPTEMBER 9, 2022

The promise of a modern data lakehouse architecture. Imagine having self-service access to all business data, anywhere it may be, and being able to explore it all at once. Imagine quickly answering burning business questions nearly instantly, without waiting for data to be found, shared, and ingested.

Architecture 102

Architecture Innovation Data Open Source 102

Netflix Tech

JUNE 20, 2024

A summary of sessions at the first Data Engineering Open Forum at Netflix on April 18th, 2024 The Data Engineering Open Forum at Netflix on April 18th, 2024. At Netflix, we aspire to entertain the world, and our data engineering teams play a crucial role in this mission by enabling data-driven decision-making at scale.

Data Engineering 82

Data Engineering Engineering Data Generative AI 82

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI.

Data 84

Data Analytics Systems Review Architecture 84

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Introduction to the Data Mesh Architecture and its Required Capabilities.

Architecture 124

Architecture Data Security Technical Review 124

Cloudera

DECEMBER 4, 2024

Our customers rely on NiFi as well as the associated sub-projects (Apache MiNiFi and Registry) to connect to structured, unstructured, and multi-modal data from a variety of data sources – from edge devices to SaaS tools to server logs and change data capture streams. Cloudera DataFlow 2.9

Metrics 93

Metrics Generative AI Open Source Data Engineering 93

TechCrunch

APRIL 13, 2021

The company is able to do this because its core architecture is somewhat different from other data pipeline and integration services that, at first glance, seem to offer a similar solution. “When you set up the connection, you aren’t taking data from Postgres and only putting it into Snowflake. Image Credits: Meroxa.

Data 248

Data Software Engineering Open Source Engineering 248

Cloudera

APRIL 14, 2021

What is Cloudera Data Engineering (CDE) ? Cloudera Data Engineering is a serverless service for Cloudera Data Platform (CDP) that allows you to submit jobs to auto-scaling virtual clusters. Refer to the following cloudera blog to understand the full potential of Cloudera Data Engineering. .

Data Engineering 84

Data Engineering Engineering Data Serverless 84

Altexsoft

APRIL 22, 2020

So, along with data scientists who create algorithms, there are data engineers, the architects of data platforms. In this article we’ll explain what a data engineer is, the field of their responsibilities, skill sets, and general role description. What is a data engineer?

Data Engineering 69

Data Engineering Engineering Artificial Inteligence Data 69

Altexsoft

AUGUST 25, 2020

Big data is tons of mixed, unstructured information that keeps piling up at high speed. That’s why traditional data transportation methods can’t efficiently manage the big data flow. Big data fosters the development of new tools for transporting, storing, and analyzing vast amounts of unstructured data.

Big Data 75

Big Data Data Engineering Engineering Data 75

CTOvision

MAY 1, 2014

The senior engineer will also take a lead role in interfacing with customer groups to understand their needs and tailor reports to enable data-driven decisions across key business areas. The senior engineer will have a great deal of freedom in choosing the right tools for the job, and will have strong support in getting it right.

Big Data 107

Big Data Data Engineering Engineering Data Center 107

CIO

JUNE 23, 2023

Not only should the data strategy be cognizant of what’s in the IT and business strategies, it should also be embedded within those strategies as well, helping them unlock even more business value for the organization.

Strategy 174

Strategy Technical Review Data Software Review 174

CIO

MAY 30, 2024

If you want to learn more about generative AI skills and tools, while also demonstrating to employers that you have the skillset to tackle generative AI projects, here are 10 certifications and certificate programs to get your started. Upon completing the learning modules, you will need to pass a chartered exam to earn the CGAI designation.

Generative AI 206

Generative AI Artificial Inteligence Programming Azure 206

O'Reilly Media - Data

APRIL 3, 2018

Data Science and Machine Learning sessions will cover tools, techniques, and case studies. This year’s sessions on Data Engineering and Architecture showcases streaming and real-time applications, along with the data platforms used at several leading companies. Data Integration and Data Pipelines sessions.

Machine Learning 231

Machine Learning Artificial Inteligence Company Case Study 231

O'Reilly Media - Ideas

FEBRUARY 18, 2020

It’s also the data source for our annual usage study, which examines the most-used topics and the top search terms. [1]. This combination of usage and search affords a contextual view that encompasses not only the tools, techniques, and technologies that members are actively using, but also the areas they’re gathering information about.

Technical Review 145

Technical Review Microservices Data Engineering Architecture 145

TechCrunch

MAY 26, 2022

Leveraging AI, Manta traces sources of data — including databases, reporting and analysis software, and modeling tools — to the ends of the pipelines that deliver it to apps and services. “Data lineage and observability are key capabilities that can solve these complex issues. billion by 2024.

Data 228

Data Weak Development Team Enterprise Analysis 228

Perficient

FEBRUARY 13, 2025

Breaking down silos has been a drumbeat of data professionals since Hadoop, but this SAP <-> Databricks initiative may help to solve one of the more intractable data engineering problems out there. SAP has a large, critical data footprint in many large enterprises. However, SAP has an opaque data model.

Government 64

Government Open Source Machine Learning Artificial Inteligence 64

CIO

JUNE 19, 2024

They may also ensure consistency in terms of processes, architecture, security, and technical governance. Our platform engineering teams, which support more than 200 applications, have innovated around automation,” says Bob Simms, former director of enterprise infrastructure delivery at the US Patent and Trademark Office (USPTO).

Weak Development Team 208

Weak Development Team Engineering UI/UX Software Development 208

Perficient

FEBRUARY 13, 2023

Introduction: We often end up creating a problem while working on data. So, here are few best practices for data engineering using snowflake: 1.Transform Using COPY and SNOWPIPE is the fastest and cheapest way to load data. In fact, this is another example of using the right tools.

Data Engineering 69

Data Engineering Engineering Data Storage 69

Apiumhub

JANUARY 26, 2022

Data engineer roles have gained significant popularity in recent years. Number of studies show that the number of data engineering job listings has increased by 50% over the year. And data science provides us with methods to make use of this data. Who are data engineers?

Data Engineering 52

Data Engineering Engineering Data Machine Learning 52

Abhishek Tiwari

FEBRUARY 26, 2020

The evolution of your technology architecture should depend on the size, culture, and skill set of your engineering organization. There are no hard-and-fast rules to figure out interdependency between technology architecture and engineering organization but below is what I think can really work well for product startup.

Architecture 82

Architecture MVC Engineering Organization 82