Connect with us

Low-code development can be used for managing big data. Here’s how to make it work for your organization.

Image: iStock/AndreyPopov

Low-code development tools are aimed at reducing application development time to market. They do this by using graphical user interfaces with point-and-click tools to cobble together applications, making it easy for end user “citizen developers” with no knowledge of programming to develop apps. Low-code tools also allow IT technical staff to insert custom code to endow the apps with functionality that the low-code tools can’t generate on their own.

SEE: Everything you need to know about using low-code platforms (free PDF) (TechRepublic)

Low-code development tools are gaining popularity in companies because users see them as ways to get around IT logjams that prevent them from getting things done.

The market concurs. By 2024, Gartner predicts that 65% of all application development will be done with low-code platforms, and that 66% of large companies will use at least four different low-code application building platforms.

Yet, low code has its limits. For example, low code is designed to work with transactional data in fixed record lengths. This makes low code a non-starter when it comes to working with unstructured big data—or does it? There are ways to use low code with big data if there is enough business value to warrant developing the methodology to facilitate it.

Here’s how it can work. Since low-code development must work with fixed records containing clearly delineated data fields within them, the major task is formatting unstructured big data into fixed records.

Here are the steps:

1. Define your business requirements

IT and business users should identify the specific business problem or use that the application is intended to address, along with types of big data that will be needed. During this step, business users, IT, and data science (if there is a separate DS department) should also identify the big data that will not be needed by the application because you don’t want to bring in any more big data than is necessary, as it creates needless overhead and bogs down processing.

2. Use AI to weed out unnecessary data

This is a task for data scientists, who will be asked to develop a data filter in the form of an artificial intelligence (AI) algorithm that will eliminate any big data not needed before the data is forwarded to the low-code application.

For example, if you are using big data weather forecasts for the Midwest, but don’t need to know the weather in Australia, the algorithm-filter can exclude any data that doesn’t pertain to the Midwest. This reduces big data file size.

3. Develop any necessary APIs

Low-code tools come with predefined APIs (application programming interfaces) to major software packages, but they don’t have APIs for every system.

In this step, IT analyzes which systems the low-code app needs to access and determines if there are any missing APIs. If an API doesn’t exist, It might be necessary to code one.

4. Convert unstructured data into fixed records

In this step, IT selects the data from the unstructured big data needed for the app, parses the unstructured data into data field chunks, and then formats the data into fixed fields within a fixed record.

SEE: Low-code platforms help with project backlogs and software development training (TechRepublic)

5. Use an ETL tool to normalize and move big data into other systems

The big data that has been formatted into fixed records must be able to match up with the data records in other systems that need to be accessed.

An extract-transform-load (ETL) tool, informed by the business rules that IT defines in it, can do this step automatically.

6. Test and refine

The final step is executing the low-code app to see if it picks up the right data, processes properly, and returns the results that the business expects.

Converting big data to run with low-code applications is time consuming but well worth it if there are enough low-code apps that can be developed off the converted big data that can be continuously reused, providing high business value to the business. 

Business value is the key, because the bottom line is always whether the data conversion effort is worth it.

Also see

Source link

0
Continue Reading

Technology

These Microsoft Azure tools can help you unlock the secrets lurking in your business data

azure synapse header

How to develop business insights from big data using Microsoft’s Azure Synapse and Azure Data Lakes technologies.

Image: Microsoft

Data lakes are an important part of a modern data analysis environment. Instead of importing all your different data sources into one data warehouse, with the complex task of building import pipelines for relational, non-relational and other data, and of trying to normalise all that data against your choice of keys, you wrap all your data in a single storage environment. On top of that storage pool, you can start to use a new generation of query tools to explore and analyse that data, working with what could be petabytes of data in real time. 

SEE: Windows 10 Start menu hacks (TechRepublic Premium)

Using data this way makes it easier to work with rapidly changing data, getting insights quickly and building reporting environments that can flag up issues as they arise. By wrapping data in one environment, you can take advantage of common access control mechanisms, applying role-based authentication and authorisation, ensuring that the right person gets access to the right data, without leaking it to the outside world. 

Working at scale with Azure Data Lake 

Using tools like Azure Active Directory and Azure Data Lake, you can significantly reduce the risk of a breach as it taps into the Microsoft Security Graph, identifying common attack patterns quickly. 

Once your data is in an Auzre Data Lake store, then you can start to run your choice of analytics tooling over it, using tools like Azure Databricks, the open-source HDInsight, or Azure’s Synapse Analytics. Working in the cloud makes sense here, as you can take advantage of large-scale Azure VM instances to build in-memory models as well as taking advantage of scalable storage to build elastic storage pools for your data lake contents. 

Microsoft recently released a second generation of Data Lake Storage, building on Azure Blobs to add disaster recovery and tiered storage to help you manage and optimise your storage costs. Azure Data Lake Storage is designed to work with gigabits of data throughput. A hierarchical namespace makes working with data easier, using directories to manage your data. And as you’re still using a data lake with many different types of data, there’s still no need for expensive and slow ETL-based transformations. 

Analysing data in Azure Synapse 

Normally you need separate analytics tooling for different types of data. If you’re building tooling to work with your own data lake, you’re often bringing together data-warehousing applications alongside big data tools, resulting in complex and often convoluted query pipelines that can be hard to document and debug. Any change in the underlying data model can be catastrophic, thanks to fragile custom analysis environments. 

Azure now offers an alternative, hybrid analytical environment in the shape of Azure Synapse Analytics, which brings together big data tooling and relational queries in a single environment by mixing SQL with Apache Spark and providing direct connections to Azure data services and to the Power Platform. It’s a combination that allows you to work at global scale while still supporting end-user visualisations and reports, and at the same time providing a platform that supports machine-learning techniques to add support for predictive analytics. 

At its heart, Synapse removes the usual barriers between standard SQL queries and big data platforms, using common metadata to work with both its own SQL dialect and Apache Spark on the same data sets, either relational tables or other stores, including CSV and JSON. It has its own import tooling that will import data into and out of data lakes, with a web-based development environment for building and exploring analytical models that go straight from data to visualisations. 

Synapse creates a data lake as part of its setup, by default using a second-generation BLOB-based instance. This hosts your data containers, in a hierarchical virtual file system. Once the data lake and associated Synapse workspace are in place, you can use the Azure Portal to open the Synapse Studio web-based development environment. 

azure-synapse-studio.jpg

Writing a PySpark query in a Spark (Scala) notebook in Azure Synapse Studio.

Image: Microsoft

Building analytical queries in Synapse Studio 

Synapse Studio is the heart of Azure Synapse Analytics, where data engineers can build and test models before deploying them in production. SQL pools manage connections to your data, using either serverless or dedicated connections. While developing models, it’s best to use the built-in serverless pool; once you’re ready to go live you can provision a dedicated pool of SQL resources that can be scaled up and down as needed. However, it’s important to remember that you’re paying for those resources even if they’re not in use. You can also set up serverless pools for Apache Spark, helping keep costs to a minimum for hybrid queries. There is some overhead when launching serverless instances, but for building reports as a batch process, that shouldn’t be an issue. 

Azure Synapse is fast: building a two-million row table takes just seconds. You can quickly work with any tabular data using familiar SQL queries, using the Studio UI to display results as charts where necessary. That same data can be loaded from your SQL store into Spark, without writing any ETL code for data conversion. All you need to do is create a new Spark notebook, and then create the database and import it from your SQL pool. Data from Spark can be passed back to the SQL pool; allowing you to use Spark to manipulate data sets for further analysis. You can use SQL queries on Spark datasets directly, simplifying what could otherwise be complex programming tasks unifying results from different platforms. 

SEE: Checklist: Securing Windows 10 systems (TechRepublic Premium)

One useful feature of Azure Data Lakes using Gen 2 storage is the ability to link to other storage accounts, allowing you to quickly work with other data sources without having to import them into your data lake store. Using Azure Synapse Studio, your queries are stored in notebooks. These notebooks can be added to pipelines to automate analysis. You can set triggers to run an analysis at set intervals, driving Power BI-based dashboards and reports. 

There’s a lot to explore with Synapse Studio, and to get the most from it requires plenty of data-engineering experience. It’s not a tool for beginners or for end users: you need to be experienced in both SQL-based data-warehousing techniques and in tools like Apache Spark. However, it’s the combination of those tools and the ability to publish results in desktop analytical tools like Power BI that makes it most useful. 

The cost of at-scale data lake analysis will always make it impossible to bring to everyone. But using a single environment to create and share analyses should go a long way towards unlocking the utility of business data. 

Also see

Source link

0
Continue Reading

Technology

Moonshots for the Treatment of Aging: Less Incrementalism, More Ambition

1 16

There is far too much incrementalism in the present research and development of therapies to treat aging. Much of the field is engaged in mimicking calorie restriction or repurposing existing drugs that were found to increase mouse life span by a few percentage points. This will not meaningfully change the shape of human life, but nonetheless costs just as much as efforts to achieve far more.

If billions of dollars and the efforts of thousands of researchers are to be devoted to initiatives to treat aging, then why not pursue the ambitious goal of rejuvenation and adding decades to healthy life spans? It is just as plausible.

Moonshots for the Treatment of Aging Less Incrementalism More Ambition

Image credit: Pixabay (Free Pixabay license)

There are just as many starting points and plausible research programs aimed at outright rejuvenation via repair of molecular damage, such as those listed in the SENS approach to aging, as there are aimed at achieving only small benefits in an aged metabolism. The heavy focus on incremental, low yield programs of research and development in the present community is frustrating, and that frustration is felt by many.

As the global population ages, there is increased interest in living longer and improving one’s quality of life in later years. However, studying aging – the decline in body function – is expensive and time-consuming. And despite research success to make model organisms live longer, there still aren’t really any feasible solutions for delaying aging in humans. With space travel, scientists and engineers couldn’t know what it would take to get to the moon. They had to extrapolate from theory and shorter-range tests. Perhaps with aging, we need a similar moonshot philosophy. Like the moon once was, we seem a long way away from provable therapies to increase human healthspan or lifespan. This review therefore focuses on radical proposals. We hope it might stimulate discussion on what we might consider doing significantly differently than ongoing aging research.

A less than encouraging sign for many of the lifespan experiments done in preclinical models, namely in mammals such as mice, is that they have modest effect sizes, often only having statistically significant effects in one of the genders, and often only in specific dietary or housing conditions. Even inhibiting one of the most potent and well-validated aging pathways, the mechanistic target of rapamycin (mTOR) pathway has arguably modest effects on lifespan – a 12-24% increase in mice. This is all to ask, if the mTOR inhibitor rapamycin is one of the potential best-case scenarios and might be predicted to have a modest effect if any (and possibly a detrimental one) in people, should it continue to receive so much focus by the aging community? Note the problems in the aging field with small and inconsistent effects for the leading strategies aren’t specific to rapamycin.

Treating individual aging-related diseases has encountered roadblocks that should also call into question whether we are on the optimal path for human aging. Alzheimer’s is a particularly well-funded and well-researched aging-related topic where there are still huge gaps in our understanding and lack of good treatment options. There has been considerable focus on amyloid beta and tau, but targeting those molecules hasn’t done much for Alzheimer’s so far, leaving many searching for answers. The point is when we spend collectively a long time on something that isn’t working well, such as manipulating a single gene or biological process, it should seem natural to consider conceptually different approaches.

Link: https://doi.org/10.3233/NHA-190064

Source: Fight Aging!




Source link

0
Continue Reading

Technology

Signal Back Up: Users May See Some Errors, Company Says Will Be Fixed in Next Update

1610920334 signal 1475682178651

Signal said it had restored its services a day after the application faced technical difficulties as it dealt with a flood of new users after rival messaging app WhatsApp announced a controversial change in privacy terms.

Signal has seen a rise in downloads following a change in WhatsApp’s privacy terms, that required WhatsApp users to share their data with both Facebook and Instagram.

Signal users might see errors in some chats as a side effect to the outage, but will be resolved in the next update of the app, the company said in a tweet.

The error does not affect the security of the chat, the company added.

The non-profit Signal Foundation based in Silicon Valley, which currently oversees the app, was launched in February 2018 with Brian Acton, who co-founded WhatsApp before selling it to Facebook, providing initial funding of $50 million (roughly Rs. 365 crores).

Signal faced a global outage that began on January 15. Although users could open the app and send messages, nothing was actually delivered.

Signal later sent Gadgets 360 a message with the following statement from its COO Aruna Harder: “We have been adding new servers and extra capacity at a record pace every single day this week, but today exceeded even our most optimistic projections. Millions upon millions of new users are sending a message that privacy matters, and we are working hard to restore service for them as quickly as possible.”

© Thomson Reuters 2021


Does WhatsApp’s new privacy policy spell the end for your privacy? We discussed this on Orbital, our weekly technology podcast, which you can subscribe to via Apple Podcasts, Google Podcasts, or RSS, download the episode, or just hit the play button below.



Source link

0
Continue Reading

Trending