Altexsoft

APRIL 22, 2020

So, along with data scientists who create algorithms, there are data engineers, the architects of data platforms. In this article we’ll explain what a data engineer is, the field of their responsibilities, skill sets, and general role description. What is a data engineer?

Data Engineering 69

Data Engineering Engineering Artificial Inteligence Data 69

O'Reilly Media - Data

APRIL 3, 2018

This year’s sessions on Data Engineering and Architecture showcases streaming and real-time applications, along with the data platforms used at several leading companies. Data Platforms sessions. Machine learning: From data preparation and integration, to model deployment and management.

Machine Learning 239

Machine Learning Artificial Inteligence Company Case Study 239

InfoQ Culture Methods

SEPTEMBER 1, 2023

Factors such as model architecture, transparency and quantization of models are required to decrease carbon emission from AI systems. There’s an increasing concern about the energy use and corresponding carbon emissions of generative AI models. By Jesse McCrosky

Generative AI 105

Generative AI Energy Architecture System 105

Xebia

MARCH 13, 2025

Please have a look at this blog post on machine learning serving architectures if you do not know the difference. Let’s say you are a Data Scientist working in a model development environment. You have complete access to all historical data. The sections below explain this in more detail.

Training 130

Training Machine Learning Artificial Inteligence Data 130

Apiumhub

JANUARY 26, 2022

Data engineer roles have gained significant popularity in recent years. Number of studies show that the number of data engineering job listings has increased by 50% over the year. And data science provides us with methods to make use of this data. Who are data engineers?

Data Engineering 52

Data Engineering Engineering Data Machine Learning 52

O'Reilly Media - Ideas

FEBRUARY 18, 2020

This year’s growth in Python usage was buoyed by its increasing popularity among data scientists and machine learning (ML) and artificial intelligence (AI) engineers. Software architecture, infrastructure, and operations are each changing rapidly. Trends in software architecture, infrastructure, and operations.

Technical Review 145

Technical Review Microservices Data Engineering Architecture 145

Confluent

JUNE 18, 2019

If you are interested in knowing more, there is a great article by Martin Kleppmann et al. As soon as the number of data points involved in your search feature increases, typically we’ll introduce a broker in between all the involved components. that describes the existing problems with heterogeneous, distributed transactions.

Scalability 97

Scalability Architecture Machine Learning Artificial Inteligence 97

Cloudera

AUGUST 30, 2022

But while state and local governments seek to improve policies, decision making, and the services constituents rely upon, data silos create accessibility and sharing challenges that hinder public sector agencies from transforming their data into a strategic asset and leveraging it for the common good. . Modern data architectures.

Architecture 79

Architecture Data Artificial Inteligence Artificial Intelligence 79

Dzone - DevOps

SEPTEMBER 6, 2024

Considering data engineering and data science, Astro and Apache Airflow rise to the top as important tools used in the management of these data workflows. This article compares Astro and Apache Airflow, explaining their architecture, features, scalability, usability, community support, and integration capabilities.

Comparison 91

Comparison Microservices Data Engineering Software Development 91

CIO

APRIL 12, 2023

After walking his executive team through the data hops, flows, integrations, and processing across different ingestion software, databases, and analytical platforms, they were shocked by the complexity of their current data architecture and technology stack. It isn’t easy.

Artificial Inteligence 218

Artificial Inteligence Artificial Intelligence Machine Learning Agile 218

InfoQ Culture Methods

JANUARY 13, 2023

This article is a summary of the 2022 software trends podcast. 2022 was another year of significant technological innovations and trends in the software industry and communities. The InfoQ podcast co-hosts met last month to discuss the major trends from 2022, and what to watch in 2023.

Trends 104

Trends Software Report Innovation 104

InfoQ Culture Methods

MAY 24, 2023

GDPR compliance should be a default feature in every application that handles PII (Personally Identifiable Information). Most organizations have an impression that GDPR is a luxury feature that needs special tools to implement.

Programming 114

Programming Compliance Applications Tools 114

CIO

NOVEMBER 29, 2022

A sea of complexity For years, data ecosystems have gotten more complex due to discrete (and not necessarily strategic) data-platform decisions aimed at addressing new projects, use cases, or initiatives. Layering technology on the overall data architecture introduces more complexity. Data and cloud strategy must align.

Organization 167

Organization Technical Review Data Artificial Inteligence 167

InfoQ Culture Methods

APRIL 7, 2020

QCon returned to London this past March for its fourteenth year in the city, attracting over 1,600 senior developers, architects, data engineers, team leads, and CTOs. This article provides a summary of the key takeaways. By Abel Avram.

Data Engineering 63

Data Engineering Engineering Data Development 63

Altexsoft

OCTOBER 24, 2019

And this data can be used to support decision making. While our brain is both the processor and the storage, companies need multiple tools to work with data. And one of the most important ones is a data warehouse. What is an Enterprise Data Warehouse? And this is what makes a data warehouse different from a Data Lake.

Architecture 50

Architecture Enterprise Data Technical Review 50

InfoQ Culture Methods

DECEMBER 28, 2020

As 2020 is coming to an end, we created this article listing some of the best posts published this year. This collection was hand-picked by nine InfoQ Editors recommending the greatest posts in their domain. It's a great piece to make sure you don't miss out on some of the InfoQ's best content.

Continuous Delivery 85

Continuous Delivery Google Cloud Data Engineering Microservices 85

TechCrunch

JANUARY 26, 2022

However, over time, as the data produced in organizations continues to expand and grow ever more complex, it has put a huge strain on organizations, both in terms of the costs of managing that data, and the investment needed to parse it in useful ways.

Analytics 218

Analytics Data Big Data Business Intelligence 218

Altexsoft

MAY 27, 2020

But, in any case, the pipeline would provide data engineers with means of managing data for training, orchestrating models, and managing them on production. Machine learning production pipeline architecture. Here we’ll look at the common architecture and the flow of such a system.

Machine Learning 89

Machine Learning Artificial Inteligence Architecture Training 89

Erik Bernhardsson

DECEMBER 15, 2020

I had my first job as a software engineer in 1999, and in the last two decades I've seen software engineering changing in ways that have made us orders of magnitude more productive. Note 1: This isn't a perfect analogy since steam power wasn't just the precious resource, it was also hard to build small steam engines.).

Software Engineering 325

Software Engineering Engineering Tools Software 325

Perficient

MARCH 23, 2023

Databricks Streaming and Apache Flink are two popular stream processing frameworks that enable developers to build real-time data pipelines, applications and services at scale. Comparison Databricks is an integrated platform for data engineering, machine learning, data science and analytics built on top of Apache Spark.

Data 83

Data Machine Learning Artificial Inteligence Data Engineering 83

InfoQ Culture Methods

JANUARY 19, 2022

In this podcast summary Thomas Betts, Wes Reisz, Shane Hastie, Charles Humble, Srini Penchikala, and Daniel Bryant discuss what they have seen in 2021 and speculate a little on what they hope to see in 2022. Topics explored included: hybrid working and the importance of ethics and sustainability within technology.

Technical Review 90

Technical Review Sustainability Technology Artificial Inteligence 90

Altexsoft

JUNE 29, 2021

This article will focus on the role of a machine learning engineer, their skills and responsibilities, and how they contribute to an AI project’s success. The role of a machine learning engineer in the data science team. Who does what in a data science team. Machine learning engineer vs. data scientist.

Artificial Inteligence 105

Artificial Inteligence Machine Learning Engineering Data Engineering 105

CIO

FEBRUARY 14, 2023

Organizations have balanced competing needs to make more efficient data-driven decisions and to build the technical infrastructure to support that goal. Many companies today struggle with legacy software applications and complex environments, which leads to difficulty in integrating new data elements or services.

Artificial Inteligence 174

Artificial Inteligence Strategy Data Machine Learning 174

CIO

DECEMBER 10, 2024

As a data-driven company, InnoGames GmbH has been exploring the opportunities (but also the legal and ethical issues) that the technology brings with it for some time. A detailed view of the KAWAII architecture. InnoGames KAWAII accesses data from our internal wiki and optionally also tickets from Jira.

Games 147

Games Artificial Inteligence Company Artificial Intelligence 147

Altexsoft

JULY 19, 2022

In the last few decades, we’ve seen a lot of architectural approaches to building data pipelines , changing one another and promising better and easier ways of deriving insights from information. There have been relational databases, data warehouses, data lakes, and even a combination of the latter two. What data mesh IS.

Architecture 52

Architecture Data Analytics Data Engineering 52

Altexsoft

JANUARY 14, 2021

Though there are countless options for storing, analyzing, and indexing data, data warehouses have remained to the point. When reviewing BI tools , we described several data warehouse tools. In this article, we’ll take a closer look at the top cloud warehouse software, including Snowflake, BigQuery, and Redshift.

Backup 115

Backup Azure Software Review Systems Review 115

Altexsoft

FEBRUARY 11, 2023

This suggests that today, there are many companies that face the need to make their data easily accessible, cleaned up, and regularly updated. Hiring a well-skilled data architect can be very helpful for that purpose. What is a data architect? What is the main difference between a data architect and a data engineer?

Data 87

Data Data Engineering Big Data Architecture 87

Altexsoft

AUGUST 21, 2019

Data obsession is all the rage today, as all businesses struggle to get data. But, unlike oil, data itself costs nothing, unless you can make sense of it. Dedicated fields of knowledge like data engineering and data science became the gold miners bringing new methods to collect, process, and store data.

Development 44

Development Software Engineering Data Engineering Architecture 44

Cloudera

JUNE 30, 2022

Today’s general availability announcement covers Iceberg running within key data services in the Cloudera Data Platform (CDP) — including Cloudera Data Warehousing ( CDW ), Cloudera Data Engineering ( CDE ), and Cloudera Machine Learning ( CML ). But the current data lakehouse architectural pattern is not enough.

Data 88

Data Analytics Machine Learning Artificial Inteligence 88

InfoWorld

MARCH 8, 2024

LinkedIn has decided to open source its data management tool, OpenHouse, which it says can help data engineers and related data infrastructure teams in an enterprise to reduce their product engineering effort and decrease the time required to deploy products or applications.

Open Source 72

Open Source Tools Data Engineering Storage 72

CIO

JULY 18, 2023

The CIO’s biggest hiring challenge is clear: “There is simply not enough talent to go around,” says Scott duFour, global CIO of business payments company Fleetcor, for whom positions in areas such as AI, cloud architecture, and data science remain the toughest to fill. Merchants Fleet also gets creative in marketing its tech roles. “We

Technical Advisors 148

Technical Advisors Artificial Inteligence Generative AI How To 148

Apiumhub

SEPTEMBER 20, 2023

In this article, we´ll be your guide to the must-attend tech conferences set to unfold in October. From software architecture to artificial intelligence and machine learning, ​​these conferences offer unparalleled insights, networking opportunities, and a glimpse into the future of technology.

Conference 98

Conference Artificial Inteligence UI/UX Machine Learning 98

Altexsoft

DECEMBER 9, 2020

Moreover, the MicroStrategy Global Analytics Study reports that access to data is extremely limited, taking 60 percent of employees hours or even days to get the information they need. To generalize and describe the basic maturity path of an organization, in this article we will use the model based on the most common one suggested by Gartner.

Analytics 102

Analytics Technical Review Technology Applications 102

Confluent

FEBRUARY 6, 2019

This blog post focuses on how the Kafka ecosystem can help solve the impedance mismatch between data scientists, data engineers and production engineers. Impedance mismatch between data scientists, data engineers and production engineers. For now, we’ll focus on Kafka.

Machine Learning 107

Machine Learning Artificial Inteligence Scalability Data Engineering 107

InfoQ Culture Methods

DECEMBER 6, 2019

As part of the 2019 end-of-year-summary content, this article collects together a list of recommended presentation recordings from the InfoQ editorial team. By Charles Humble, Ben Linders, Arthur Casals, Manuel Pais, Erik Costlow, Shane Hastie, Shaaron A Alvares, Srini Penchikala, Michael Redlich, Steef-Jan Wiggers, Daniel Bryant.

Data Engineering 52

Data Engineering Architecture Programming Engineering 52

Altexsoft

NOVEMBER 10, 2021

The pun being obvious, there’s more to that than just a new term: Data lakehouses combine the best features of both data lakes and data warehouses and this post will explain this all. What is a data lakehouse? Traditional data warehouse platform architecture. Data lake architecture example.

Architecture 52

Architecture Data Machine Learning Storage 52

Altexsoft

AUGUST 29, 2023

Data lakes emerged as expansive reservoirs where raw data in its most natural state could commingle freely, offering unprecedented flexibility and scalability. This article explains what a data lake is, its architecture, and diverse use cases. Watch our video explaining how data engineering works.

Architecture 52

Architecture Data Storage Machine Learning 52