java - Found multiple defaults.yaml resources

Question

when i tried to submit the topology i found this

Exception in thread "main" java.lang.RuntimeException: Found multiple defaults.yaml resources. You're probably bundling the Storm jars with your topology jar.
at backtype.storm.utils.Utils.findAndReadConfigFile(Utils.java:115)
at backtype.storm.utils.Utils.readDefaultConfig(Utils.java:135)
at backtype.storm.utils.Utils.readStormConfig(Utils.java:155)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:61)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:40)
at trident.myproject.main(myproject.java:288)

But this error appeared after updated in pom.xml by

<scope>compile</scope> instead of <scope>provided</scope>

that because i was an error

An exception occured while executing the Java class. storm/trident/state/StateFactory

here pom file

<plugins>
    <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
            <descriptorRefs>
                <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
            <archive>
                <manifest>
                    <mainClass>trident.myproject</mainClass>
                    <!-- <mainClass>crawler.Crawler</mainClass> -->
                </manifest>
            </archive>
        </configuration>

part 2 of pom file

<executions>
    <execution>
        <id>make-assembly</id>
        <phase>package</phase>
        <goals>
            <goal>single</goal>
        </goals>
    </execution>
</executions>

part 3 of pom file

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
        <configuration>
            <source>1.6</source>
            <target>1.6</target>
        </configuration>
    </plugin>
</plugins>

Answer 1

There is a fundamental difference in running a topology in LocalCluster or remotely via StormSubmitter (which is the default setting in the project).

The scope of storm-core is set to <scope>provided</scope> be the default because those class files are available in the cluster anyway. provided tells maven, that those classes must not be included in the jar file that is assembled, thus reducing the size of the jar. Furthermore, this avoids conflicts if files are provided multiple times -- that is what happens with default.yaml if you change the scope to compile. For those case, all files from storm-core are packaged into you jar and submitted to the cluster. Storm finds the file defaults.yaml "locally" (ie, locally on the worker machine in the cluster) and in your jar. Thus, Storm does not know which one to use and raises an error.

However, provided excludes those class files if you run locally, too. Of course, locally those files are not available automatically but must be included in CLASSPATH when starting up the local JVM. As provided excludes the files from storm-core you get the ClassNotFound exception.

As an alternative to change the scope each time you want to submit to a different environment, you can set the scope to compile and include your topology Main/Bolt/Spout classes explicitly in your maven-jar-plugin settings. This explicit inclusion automatically excludes all other files from the jar, ie, all files from storm-core.

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-jar-plugin</artifactId>
  <version>2.6</version>

  <executions>
    <execution>
      <id>MyTopology</id>
      <phase>package</phase>
      <goals>
        <goal>jar</goal>
      </goals>
      <configuration>
        <includes>
          <include>my/topology/package/**/*.class</include>
        </includes>
      </configuration>
    </execution>
  </executions>
</plugin>