Pig is an Apache open-source project that runs on Hadoop and provides a parallel data flow engine. It contains the pig Latin language, which is used to express data flow. It includes actions such as sorting, joining, filtering, and scripting UDF (User Defined Functions) for reading, writing, and processing. Pig stores and processes the entire task using Map Reduce and HDFS.
Ques. 1): What benefits does Pig have over MapReduce?
Answer:
The development cycle for MapReduce is extremely long. It takes a long time to write mappers and reducers, compile and package the code, submit tasks, and retrieve the results. Dataset joins are quite complex to perform. Low level and stiff, resulting in a large amount of specialised user code that is difficult to maintain and reuse is difficult.
Pig does not require the compilation or packaging of code. Pig operators will be turned into maps or jobs will be reduced internally. Pig Latin supports all common data-processing procedures, as well as high-level abstraction for processing big data sets.
Ques. 2): Is Piglatin a Typographically Strong Language? If so, how did you arrive at your conclusion?
Answer:
In a strongly typed language, the type of all variables must be declared up front. When you explain the schema of the data in Apache Pig, it expects the data to be in the same format.
When the schema is unknown, however, the script will adjust to the actual data types at runtime. PigLatin can thus be described as firmly typed in most circumstances but gently typed in others, i.e. it continues to work with data that does not meet its expectations.
Ques. 3): What are Pig's disadvantages?
Answer:
Pig has a number of flaws, including:
Pig isn't the best choice for real-time applications.
When you need to get a single record from a large dataset, Pig isn't very useful.
It works in batches since it uses MapReduce.
Ques. 4): What is Pig Storage, exactly?
Answer:
Pig comes with a default load function called Pig Storage. Additionally, we may use pig storage to import data from a file system into the pig.
While loading data into pig storage, we may also provide the data delimiter (how the fields in the record are separated). We can also provide the data's schema as well as the data's type.
Ques. 5): Explain Grunt in Pig and its characteristics.
Answer:
The Grunt takes on the role of an Interactive Shell Pig. Grunt's main characteristics are:
To move the cursor to the end of a line, press the ctrl-e key combination.
As a Grunt retains command history, the lines in the history buffer can be recalled using the up and down cursor keys.
Grunt supports the auto-completion method by attempting to finish Pig Latin keywords and functions when the Tab key is hit.
Ques. 6): What Does Pig Flatten Mean?
Answer:
When there is data in a tuple or a bag, we may use the Flatten modifier in Pig to remove the level of nesting from that data. Un-nests bags and tuples should be flattened. The Flatten operation for tuples will substitute the fields of a tuple for a tuple, however un-nesting bags is a little more complicated because it necessitates the creation of new tuples.
Ques. 7): Can you distinguish between logical and physical plans?
Answer:
Pig goes through a few processes while converting a Pig Latin Script into MapReduce jobs. Pig generates a logical plan after performing basic parsing and semantic testing. Pig's logical plan, which is executed during execution, describes the logical operators. Pig then generates a physical plan. The physical plan specifies the physical operators required to execute the script.
Ques. 8): In Pig, what does a co-group do?
Answer:
Co-group unites the data collection by grouping only one of the data sets. It then groups the elements by their common field and provides a set of records with two distinct bags. The records of the first data set with the common data set are in the first bag, and the records of the second data set with the same data set are in the second bag.
Ques. 9): Explain the bag.
Answer:
Pig includes several data models, including a bag. The bag is an unorganised collection of tuples with possibly duplicates that is used to store collections while they are being grouped. The size of the bag is equal to the size of the local disc, implying that the bag's size is limited. When the bag is full, Pig will empty it onto the local disc and only maintain a portion of it in memory. It is not necessary for the entire bag to fit into memory. With ", we signify bags.
Ques. 10): Can you describe the similarities and differences between Pig and Hive?
Answer:
Both Hive and Pig have similar characteristics.
Both internally transform the commands to MapReduce.
High-level abstractions are provided by both technologies.
Low-latency queries are not supported by either.
OLAP and OLTP are not supported by either.
Ques. 11): How do Apache Pig and SQL compare?
Answer:
The use of Apache Pig for ETL, lazy evaluation, storing data at any stage in the pipeline, support for pipeline splits, and explicit specification of execution plans set it apart from SQL. SQL is built around queries that return only one result. SQL doesn't have a built-in mechanism for separating a data processing stream into sub-streams and applying various operators to each one.
User code can be added at any step in the pipeline with Apache Pig, whereas with SQL, data must first be put into the database before the cleaning and transformation process can begin.
Ques. 12): Can Apache Pig Scripts Join Multiple Fields?
Answer:
Yes, several fields can be joined in PIG scripts since join procedures take records from one input and combine them with records from another. This is accomplished by specifying the keys for each input and joining the two rows when the keys are equal.
Ques. 13): What is the difference between the commands store and dumps?
Answer:
After running the dump command, the data appears on the console, but it is not saved. Whereas the output is executed in a folder and the store is stored in the local file system or HDFS. Most hadoop developers utilised the'store' command to store data in HDFS in a protected environment.
Ques. 14): Is 'FUNCTIONAL' a User Defined Function (UDF)?
Answer:
No, the keyword 'FUNCTIONAL' does not represent a User Defined Function (UDF). Some functions must be overridden while using UDF. You must certainly complete your tasks using only these functions. However, because the keyword 'FUNCTIONAL' is a built-in function (a pre-defined function), it cannot be used as a UDF.
Ques. 15): Which method must be overridden when writing evaluate UDF?
Answer:
When developing UDF in Pig, we must override the method exec(). While the base class may change, when developing filter UDF, we must extend FilterFunc, and when writing evaluate UDF, we must extend EvalFunc. EvaluFunc is parameterized, and the return type must be specified as well.
Ques. 16): What role does MapReduce play in Pig programming?
Answer:
Pig is a high-level framework that simplifies the execution of various Hadoop data analysis problems. A Pig Latin programme is similar to a SQL query that is executed using an execution engine. The Pig engine can convert programmes into MapReduce jobs, with MapReduce serving as the execution engine.
Ques. 17): What Debugging Tools Are Available For Apache Pig Scripts?
Answer:
The essential debugging utilities in Apache Pig are describe and explain.
When trying to troubleshoot or optimise PigLatin scripts, Hadoop developers will find the explain function useful. In the grunt interactive shell, explain can be applied to a specific alias in the script or to the entire script. The explain programme generates multiple text-based graphs that can be printed to a file.
When building Pig scripts, the describe debugging utility is useful since it displays the schema of a relation in the script. Beginners learning Apache Pig can use the describe utility to see how each operator alters data. A pig script can have multiple describes.
Ques. 18): What are the relation operations in Pig? Explain any two with examples.
Answer:
The relational operations in Pig:
foreach, order by, filters, group, distinct, join, limit.foreach: It takes a set of expressions and applies them to all records in the data pipeline to the next operator.A =LOAD ‘input’ as (emp_name :charrarray, emp_id : long, emp_add : chararray, phone : chararray, preferences : map [] );B = foreach A generate emp_name, emp_id;Filters: It contains a predicate and it allows us to select which records will be retained in our data pipeline.
Syntax: alias = FILTER alias BY expression;
Alias indicates the name of the relation, By indicates required keyword and the expression has Boolean.
Example: M = FILTER N BY F5 == 50;
Ques. 19): What are some Apache Pig use cases that come to mind?
Answer:
The Apache Pig large data tools are used for iterative processing, raw data exploration, and standard ETL data pipelines. Pig is commonly used by researchers who want to use the data before it is cleansed and placed into the data warehouse because it can operate in situations where the schema is unknown, inconsistent, or incomplete.
It can be used by a website to track the response of users to various sorts of adverts, photos, articles, and so on in order to construct behaviour prediction models.
Ques. 20): In Apache Pig, what is the purpose of illustrating?
Answer:
Illustrate is used to run Pig scripts on large datasets, which might take a long time. That is why developers run pig scripts on sample data, even though it is probable that the sample data selected will not execute the script correctly. If the script includes a join operator, for example, there must be a small number of records in the sample data with the same key, or the join operation will fail. Developers manage these issues by using the function illustrate, which takes data from the sample and ensures that some records pass through while others are restricted by modifying records in such a way that they follow the condition set whenever it encounters operators like the filter or join, which remove data. Illustrate displays each step's output but does not run MapReduce operations.
No comments:
Post a Comment