The Download: Tech Talks by the HPCC Systems Community, Episode 10

Join Anirudh Shah, Founder & CEO, 3LOQ Labs, and Flavio Villanustre, VP Technology, HPCC Systems, to learn how 3LOQ is solving the problem of customer churn with open source big data and machine learning technology. 3LOQ addresses this challenge by deploying proprietary machine learning algorithms to analyze billions of data points and map out dynamic feature recommendations to reinforce repeated usage of a product. The end result? Reduced churn with high customer engagement for businesses.

3LOQ recently partnered with a leading Indian banking institution to increase adoption of their digital channels. The project yielded impressive results for the client, including a:

· 45% reduction in customer churn
· 145% increase in digital banking transactions
· 75% increase in users who made four or more transactions per month

In this webcast, Flavio will give an overview of one of the key tech tools that contributes to 3LOQ's success, the completely free, open source HPCC Systems big data platform. Anirudh will share how 3LOQ Labs leverages this platform to:

• Analyze four terabytes of data combined with built-in analytics libraries to create personalized recommendations
• Utilize efficient coding in an implicitly parallel platform that allows prototypes to be developed and iterated quickly
• Enable horizontal scaling on commodity hardware, with the flexibility to deploy both on premises and in the cloud

Join us as we continue this series of webinars specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community. This episode will feature three speakers on the following topics:

Tai Donovan, Robotics Director, American Heritage School - High School Autonomous Agricultural Project
A group of 5-6 students are working on an autonomous agricultural project with the goal of providing time sensitive data to the owner-operator/farmer/grower of a production farm. Tai will discuss their challenges and how he is using HPCC Systems.

Lorraine Chapman, Consulting Business Analyst, LexisNexis Risk Solutions - Meet Our Summer Interns
By the end of 2018, ten students will have completed projects as part of the HPCC Systems intern program. Find out about these students, including where and what they are studying, the projects they will be working on and the intern experience we provide to help them feel part of the team. Lorraine will also speak about how you can get involved with the program by being a mentor, or contributing a project idea for a new feature or enhancement to the HPCC Systems platform and/or Machine Learning Library.

Richard Taylor, Chief Trainer, HPCC Systems, LexisNexis Risk Solutions – Current/Longest Event Sequence by Month
Richard will discuss processing event dates to discover for each event within a given time frame: the current number of sequential months the event occurred, and the longest contiguous month-by-month sequence. This topic is based on questions from one of our Statistical Modelers (new to ECL) regarding how to approach the problem in a non-procedural manner. The example code will make use of the GROUP and HAVING functions.

Join us as we continue this series of webinars specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Episode 13 includes Tech Talks featuring speakers from our community on topics covering the Future of Automotive Telemetry: Assessing Autonomous Vehicle Risk Implications using Simulated Data, Developing A Custom, Pluggable HPCC Systems Security Manager and Understanding the ECL Watch Graphs. View the full details at hpccsystems.com

Join us as we continue this series of webinars specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Episode 12 includes Tech Talks featuring speakers from our community on topics covering exploratory data analysis, geospatial solutions and ECL Tips leveraging the HPCC Systems platform.


1) Itauma Itauma, PhD Candidate, Keiser University - Conducting exploratory data analysis in educational research using HPCC Systems®

2) Ignacio Calvo, LexisNexis Risk Solutions - Big Data and Geospatial with HPCC Systems®

3) Bob Foreman, Senior Software Engineer, HPCC Systems, LexisNexis Risk Solutions - ECL Tip of the Month

Over 4,000 U.S. workers die on the job every year. While new wearable technologies are aggressively entering consumer applications, industrial safety equipment has not seen a fundamental innovation in the last decade.

Join us to learn how Guardhat CTO Anupam Sengupta and Guardhat use open source big data technology to address this issue with its “smart hard hat ecosystem”, an industrial wearable that uses IoT and wireless communications systems to protect and empower industrial workers.

In this webcast, Flavio will give an overview of the completely free, open source HPCC Systems big data platform.

Anupam will share how Guardhat leveraged this platform to:
• Allow real-time complex event processing of vast amounts of streaming data.
• Enable horizontal scaling on commodity hardware, with the flexibility to deploy both on premises and in the cloud.
• Support big data analytics including the ability to analyze, identify, and predict trends.
• Enable rapid green-field development

Join us as we continue this series of webinars specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Episode 11 includes Tech Talks featuring speakers from our community on topics covering Big Data solutions, Spark Integration and other ECL Tips leveraging the HPCC Systems platform.

1) Raj Chandrasekaran, CTO & Co-Founder, ClearFunnel - Scaling Data Science capabilities: Leveraging a homogeneous Big Data ecosystem

2) James McMullan, Software Engineer III, LexisNexis Risk Solutions - HDFS Connector Preview

3) Bob Foreman, Senior Software Engineer, LexisNexis Risk Solutions - Building a RELATIONal Dataset - A Valentine’s Day Special!

Join us as we continue this series of webinars specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Episode 10 will kick off our first Tech Talk in 2018 and includes 15 minute Tech Talks featuring speakers from the community:

1) Chris Gropp, PhD candidate, Clemson University - Asking the Right Questions with Machine Learning

The HPCC Systems Machine Learning Library contains a number of powerful tools, but it is important to use them properly. Chris will discuss how to ask the right questions by taking a step backwards from the methods themselves and examining the requirements defined by the applications.

2) Rodrigo Pastrana, Software Architect, LexisNexis Risk Solutions - Creating Front-facing Web Services to Deliver your HPCC Systems Query Data

The HPCC Systems platform provides everything you need to easily create production grade web services to deliver your query data. Rodrigo will discuss the tools and frameworks provided by the HPCC Systems platform and walk through the end-to-end creation of a sample web service.

3) Richard Taylor, Chief Trainer, HPCC Systems, LexisNexis Risk Solutions – ECL Tips and Cool Tricks

Join Richard for the latest tips and tricks with using ECL! In this session, he will talk about the PARSE function and interesting techniques used in data parsing.

Bringing heterogeneous data into a homogenous data warehouse environment is one of the most daunting aspects of any big data implementation.

Even though Apache Spark and HPCC Systems Thor can be thought of as complementary, there is interest in comparing their performance with data analytics-related benchmarks, specifically transformation, cleaning, normalization, and aggregation. Join us to hear how HPCC Systems Thor's performance compares to Apache Spark utilizing standard benchmarking methodologies.

Learn how these benchmarks and HPCC Systems can help you establish new baselines that:
•Improve the speed and accuracy of the transformation, cleaning, normalization, and aggregation processes
•Enable efficient use of developer resources and development budgets
•Facilitate the use of standard hardware, operating systems, and protocols

As farmers grapple with how they will feed an increasing global population, the need to harness data and analytics has become more critical. Changing diets, demand for healthier food options, and decreasing water availability are just some of the challenges that face agriculture today. Combine that with global market volatility and rising input costs such as water, chemicals, seeds, and more and it is harder than ever for farmers to be profitable and sustainable.

As Proagrica searched for ways to help the agriculture industry use data-driven decision making for crops and livestock production, they decided to adopt HPCC Systems® as their big data partner. HPCC Systems not only delivered a scalable, resilient and secure platform, but it also met their projected future expansion needs.

Join Jeff Bradshaw, Group CTO for Adaptris within Proagrica and Flavio Villanustre, Vice President of Technology for HPCC Systems, as they discuss how they implemented HPCC Systems and the Adaptris enterprise service bus to incorporate massive, diverse data sets from within complex, secure environments. Jeff and Flavio will share insights on best implementation practices to deliver clean/quality data, provide data security, and deliver the real-time analytics their customers demanded.

In this webcast, Proagrica will share their mission and their experiences with:

· The open source, end to end big data processing platform, HPCC Systems
· Rapid development with a growing set of real time data sources
· Seamlessly integrating with existing infrastructure to simplify deployment and management capabilities
· Real-time data ingestion and flexible stream processing from massively diverse data sets
· Enabling real-time analytics while providing for data security

Join us November 16 for another episode of The Download: HPCC Systems Community Tech Talks!

This series of workshops is specifically designed for the community by the community with the goal to share knowledge, spark innovation, and further build and link the relationships within our HPCC Systems community.

Featured speakers and topics include:

Robert Pelley, Architect for UK and Ireland, LexisNexis Risk Solutions - Integrating REDIS with HPCC Systems in high volume UK infrastructure.
REDIS (REmote DIctionary Server), an in-memory caching technology will be integrated into the LexisNexis application stack to provide an additional cache at the entry point of the application infrastructure. This will serve as a cache of Product Responses whereas existing caches will continue to serve Vendor Responses. The aim of the REDIS front-end Product Response cache is to improve system throughput and response times.

Bob Foreman, Senior Software Engineer, LexisNexis Risk Solutions – ECL Tips: The Bright Green Data Generation Machine (DataGen)
Bob will be talking about the ECL Code Generator, DataGen, best practices and tips for its use in generating random data and will walk through a short demo.

This series of workshops is specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Featured speakers and topics include:

Xiaoming Wang (Ming), Consultant Software Engineer, LexisNexis Risk Solutions - Initial HPCC Systems integration with Jupyter Notebook
The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more. In this talk, Ming will give an overview on plans for implementing a typescript based ECL kernel utilizing HPCC Systems JavaScript libraries to submit ECL code and return Workunit results rendered in Jupyter Notebook cells.

Bob Foreman, Senior Software Engineer, LexisNexis Risk Solutions – ECL Tips: PROCESS and AGGREGATE transform functions
Have you ever wanted to expand the power of your ECL ITERATE and ROLLUP statements? Bob Foreman discusses the next level PROCESS and AGGREGATE transform functions, and illustrates practical examples that were shown in our HPCC Systems forums.

Join us as we continue this series of workshops specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Episode 6 will include 15 minute Tech Talks from our HPCC Systems summer interns. Come hear the cool projects they have been working on!

This session will include Lorraine Chapman, Consulting Business Analyst, LexisNexis Risk Solutions, who manages our HPCC Systems internship program, along with our 2017 HPCC Systems summer interns. Come hear the cool projects they have been working on!

Lily Xu - PhD student of Computer Science at Clemson University, USA
Extending the YinYang K-Means machine learning algorithm in ECL

Vivek Nair - PhD student of Computer Science North Carolina State University, USA
Working to allow Spark to use HPCC Systems as a datastore and provide ECL programmers with the ability to access Spark algorithms.

George Mathew - PhD student of Computer Science at North Carolina State University, USA
Implementing the Gradient Trees machine learning algorithm in ECL to build predictive models.

This series of workshops is specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Our featured speakers and topics include:

1. Jeff Bradshaw, CTO, Adaptris - Interlok Deep Dive

In this talk, Jeff will explain how Interlok is used within the HPCC Systems platform, specifically the Thor component, and developing entity models for delivering data insights.

2. Jon Burger, Sr Architect, LexisNexis Risk Solutions - Hive360, Cloud Ported HPCC Systems Platform

HPCC Systems is excited to announce the creation of the Hive360 & Swarm360 stacks. Hive360 and its companion Swarm360 are a set of AWS cloud formation scripts designed to easily create a scalable, self-configuring, self-healing, on-demand HPCC platform within an existing AWS VPC. This talk will introduce you to Hive360 and its components, give a brief demonstration of the process and answer any questions you have about this technology.

3. Rodrigo Pastrana, Software Architect, LexisNexis Risk Solutions - SQL on HPCC Systems

HPCC Systems provides a SQL interface into its data files and published ROXIE queries called WsSQL. As its name implies, this functionality is provided as a web service, which allows interactive and/or programmatic SQL based access to HPCC Systems.

4. Bob Foreman, Senior Software Engineer, HPCC Systems, LexisNexis Risk Solutions - ECL Tip of the Month

This session will showcase an “ECL Tip of the Month”, presented by one of our ECL instructors, Bob Foreman. The tip will usually be something interesting that was posted on our HPCC Systems Support Forums, or a cool teaching example found in one of our many ECL classes.

This series of workshops is specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

Our featured speakers and topics include:

Gordon Smith, Lead Architect, LexisNexis Risk Solutions - “Visualizer” – the ECL Bundle
John Holt, Lead Architect, LexisNexis Risk Solutions – An Update of the Machine Learning Bundles
David de Hilster, Consulting Software Engineer, LexisNexis Risk Solutions - The ECL IDE Goes Multi-Language – Computer Languages that Is!

Jessica Lorti, Director Marketing, LexisNexis Risk Solutions - HPCC Systems Marketing & Web Site Update

This series of workshops is specifically designed for the community by the community with the goal to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

1. Joselito (Joey) Chua , PhD, Manager Software Engineer, Optimal Decisions Group - Prescriptive Analytics - a Software Engineering Perspective

This talk presents an overview of prescriptive techniques involving simulation and optimisation, the engineering challenges in building prescriptive tools, and HPCC solutions for those challenges.

2. Jill Luber, Senior Architect, LexisNexis Risk Solutions - Migrating an ECL code repository into Git, Part II

This session will take a quick look at a migration plan that moved ECL production code, production processes and developers out of MySQL/SVN and into a Git code management culture. This includes migrating both ROXIE and Thor processes to use Git branches across multiple HPCC Systems environments, all while continuing production data builds and releases.

3. Michael Gardner, Software Engineer II, LexisNexis Risk Solutions - HPCC Systems Platform: Java APIs and tools

This presentation will be in regards to the Java API and tools released by the HPCC Systems Platform team. These projects include wsclient, rdf2hpcc, clienttools, and jdbc. These open source projects, which can be found in the hpcc-systems github repositories, are designed to allow downstream developers a consistent means by which to interface with the HPCC Systems Platform, and to facilitate the workflow of common tasks a downstream developer might be concerned with.

4. Bob Foreman, Senior Software Engineer, HPCC Systems, LexisNexis Risk Solutions - In Search of the Lost Tutorial – the best ECL lesson you have never seen.

In this presentation, Bob will explore David Bayliss’ ECL Bible Tutorial, with particular focus on the GRAPH function and building the inverted index for the ROXIE search.

The purpose of the workshop will be to share knowledge, spark innovation and further build and link the relationships within our HPCC Systems community.

1. Fujio Turner, Solutions Architect, Couchbase - Mobile/IoT & HPCC Systems

​Fujio will discuss the challenges around IoT and address the following questions:

As there are more mobile and embedded devices all generating more data, what does that mean now and for the future?
What has to change in an organization's infrastructure to keep up?
And how can I best take advantage this new stream of information?

2. Jacob Pellock, Sr Director Software Engineering, LexisNexis - Operationalizing jobs on Thor utilizing Python, Git and HPCC Systems client tools - Part I

So you’ve setup your HPCC Systems cluster and you’ve written your ECL code. Now you want to take the ECL you’ve written into production. Jacob will explain what technologies we’ve leveraged in bringing our LexisNexis data warehouse into production.

3. Roger Dev, Sr Architect, LexisNexis - Basic Linear Algebra Subsystem (BLAS) and Parallel Block BLAS (PBBlas) libraries for HPCC Systems.

Manipulation of matrix data via Linear Algebra operations lies at the heart of many data-mining and machine-learning techniques. New modules for HPCC provide highly scalable and performant implementations of these operations. BLAS provides an industry-standardized set of highly-optimized linear algebra operations. PBBlas extends these operations to mega-scale, splitting the operations into parallelizable units that can be balanced across an HPCC cluster. This talk provides an introduction to BLAS, describes the techniques and features of PBBlas, and provides an overview of the PBBlas interface.

4. Richard Taylor, Chief Trainer, HPCC Systems - HPCC Systems Training: Updates and Deep Dives on Cool Code

Richard will be presenting an update on what’s going on with ECL/HPCC/SALT/KEL training courses. He will also be selecting some interesting code snippets.

Introducing The Download: Tech Talks by the HPCC Systems community!

We had a great lineup of speakers for our first Tech Talk on January 12:

Flavio Villanustre, VP Technology, LexisNexis
Anirudh Shah, Co-Founder, 3Loq
Allan Wrobel, Sr Software Engineer, LexisNexis
Lorraine Chapman, Consulting Business Analyst, HPCC Systems
Detailed Agenda - Watch Recording

Flavio Villanustre, VP Technology, LexisNexis
Welcome and HPCC Systems Update

Anirudh Shah, Co-Founder, 3Loq
How we use HPCC Systems to process more than 500 monthly marketing campaigns at the largest private bank in India across the banks entire portfolio.
Our experience with HPCC Systems in production
Automation and data sanity frameworks

Allan Wrobel, Senior Engineer, LexisNexis
Making full use of Superfiles to make order of magnitude improvements to build times on THOR. (plus fringe benefits)
Thor is well known for making short the processing of billions of records, and this promotes the tendency to use brute force in its deployment. Watch how the UK managed to implement efficiency over brute force to reduce the processing time for a daily build of a billion record ingest file from 12 hours, to 2 hours, and enabled further speed increases in other processes.

Lorraine Chapman, Consulting Business Analyst, HPCC Systems
In 2015, HPCC Systems was an accepted organization for Google Summer of Code (GSoC) taking on 2 students involved in this program. However, we had the bandwidth to support more students and so the HPCC Systems summer internship program was born. Four students joined the program in 2015 and four more in 2016. We will apply for GSoC and run our intern program again in 2017. Hear how the programs work, how projects are identified and find out about student successes on these programs.